| Re: [hono-dev] Migration to Prometheus and metrics prefixes |
Well I was assuming that we drop InfluxDB and go for Prometheus. Not having any dual stack approach.
Sure, we can also do a Spring Boot upgrade first. I only think that we shouldn't do this in one big PR.So I think the following topics need some decision:1) Do we want to upgrade to Spring Boot 2.x now? Before or after?2) Do we need to keep the current InfluxDB and its data format alive? If yes, then should we add a deprecation note to the Hono 0.7 release?3) Do we want to switch to Micrometer? (I assume Spring Boot 2 would also still support Dropwizard).
_______________________________________________On Wed, Jun 6, 2018 at 4:14 PM, Marc Pellmann <pellmann@xxxxxxxxx> wrote:Biggest problem for users of Hono right now might be, if they need to create new queries for their Grafana boards. But when we want the change to Prometheus and the users want to also go with this change they need to do this anyway.And if they do want to stay (for some time..) with InfluxDB it seems to be possible to remap the tags to the old hierarchical structure again and write it with the graphite protocol as before (https://micrometer.io/docs/registry/graphite).. but it would be a funny mapmap approach..On Wed, Jun 6, 2018 at 2:41 PM Jens Reimann <jreimann@xxxxxxxxxx> wrote:Hm … I do like this approach … but it make the current task a lot bigger. However it also looks like there is better integration with all components. There is support for Prometheus out of the box and support for vertx as well.But if we want to do this in the near future, then right now might be the best timing. As we need to re-do the Graphana dashboards in any case. Doing it twice doesn't sound like a good idea.So if everyone is on board with this idea, I would like give that a try._______________________________________________On Wed, Jun 6, 2018 at 1:58 PM, Marc Pellmann <pellmann@xxxxxxxxx> wrote:Hi Jens,we use the "traditional"/hierarchical naming of the metrics like x.y.z. Dropwizard (the used framework) typically does it this way. In newer versions or at newer frameworks a tag bases approach seams more often used.In our example we use templates to map some of the parts (like the y) to tags, since InfluxDB also like tags more. In this way the "prefix" could be mapped to a tag.But maybe we should consider to change our approach to tag-based/dimensional metrics at all?!In Spring Boot 2 they changed the default framework from Dropwizard to Micrometer which is tag based [1]. There is a backport for Spring Boot 1.x with only one added dependency - so we could just change this part.Maybe we should take a look?MarcOn Wed, Jun 6, 2018 at 1:18 PM Jens Reimann <jreimann@xxxxxxxxxx> wrote:_______________________________________________Hi everyone,I am currently working on issue #642 [1] and have a first version with Prometheus running. Now I do have the following issue and would like to discuss how to best approach this.I did enable metrics not only for the HTTP, MQTT and Messaging, but also for the device registry and the auth service. This allows to get metrics about the JVM (memory, threads, …) and vertx which to me would be interesting as well.However, currently Hono does set a prefix to metrics. The default being "hono", which will be overridden by some application (e.g. "hono.mqtt").Now this results in the same information reported in different ways. Just looking at "jvm.memory.total.used":The Auth & Device service report it (e.g.) as:---hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-service-device-registry",instance="172.17.0.15:8081",job="kubernetes-service-endpoints",kubernetes_name="hono-service-device-registry",kubernetes_namespace="hono"}
hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-service-auth",instance="172.17.0.17:8081",job="kubernetes-service-endpoints",kubernetes_name="hono-service-auth",kubernetes_namespace="hono"}---Which means that with a simple Prometheus query of "hono_jvm_memory_total_used" you can get the JVM memory usage from two applications.If you want a specific application you need to filter: hono_jvm_memory_total_used{deploymentconfig="hono-service-auth"}Now if you want the same information from the HTTP adapter you will need to use: hono_http_jvm_memory_heap_usage---hono_http_jvm_memory_heap_usage{app="hono-adapter",deploymentconfig="hono-adapter-http-vertx",instance="172.17.0.16:8081",job="kubernetes-service-endpoints",kubernetes_name="hono-adapter-http-vertx",kubernetes_namespace="hono"}---I do think that inconsistency should be resolved. There are two ways to do this, set a custom prefix for all Hono applications. Use the same prefix for all Hono applications.Kubernetes itself does not use a prefix at all - query: go_memstats_alloc_bytes … results in:---go_memstats_alloc_bytes{instance="192.168.122.20:8443",job="kubernetes-apiservers"}
go_memstats_alloc_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="localhost",job="kubernetes-nodes",kubernetes_io_hostname="localhost"}
go_memstats_alloc_bytes{app="prometheus",instance="172.17.0.13:8443",job="kubernetes-service-endpoints",kubernetes_name="prometheus",kubernetes_namespace="kube-system",name="prometheus"}
go_memstats_alloc_bytes{app="openshift-web-console",instance="172.17.0.3:8443",job="kubernetes-service-endpoints",kubernetes_name="webconsole",kubernetes_namespace="openshift-web-console"}---There is no application specific prefix and proper tagging is used.So my proposal would be to completely drop the prefix as I don't think it adds much and only makes things more complex. Instead using tagging of the values (like seen above).A compromise could be, although I would like to avoid this, to make the prefix configurable, and allowing an empty prefix in the process.Let me know what you think.CheersJens
--Jens Reimann
Senior Software Engineer / EMEA ENG Middleware
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany
phone: +49 89 2050 71286
_____________________________________________________________________________
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill
hono-dev mailing list
hono-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/hono-dev
_______________________________________________
hono-dev mailing list
hono-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/hono-dev
--Jens Reimann
Senior Software Engineer / EMEA ENG Middleware
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany
phone: +49 89 2050 71286
_____________________________________________________________________________
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill
hono-dev mailing list
hono-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/hono-dev
_______________________________________________
hono-dev mailing list
hono-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/hono-dev
--Jens Reimann
Senior Software Engineer / EMEA ENG Middleware
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany
phone: +49 89 2050 71286
_____________________________________________________________________________
Red Hat GmbH, www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill
hono-dev mailing list
hono-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/hono-dev