Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [hono-dev] Migration to Prometheus and metrics prefixes 
On Wed, 2018-06-06 at 14:41 +0200, Jens Reimann wrote:
> Hm … I do like this approach … but it make the current task a lot bigger.
> However it also looks like there is better integration with all components.
> There is support for Prometheus out of the box and support for vertx as well.
> 
> But if we want to do this in the near future, then right now might be the best
> timing. As we need to re-do the Graphana dashboards in any case. Doing it twice
> doesn't sound like a good idea.
> 
> So if everyone is on board with this idea, I would like give that a try. 
> 

Meaning: Using the back ported Micrometer based metrics from Spring Boot 2.0?

> 
> 
> On Wed, Jun 6, 2018 at 1:58 PM, Marc Pellmann <pellmann@xxxxxxxxx> wrote:
> > Hi Jens,
> > 
> > we use the "traditional"/hierarchical naming of the metrics like x.y.z.
> > Dropwizard (the used framework) typically does it this way. In newer versions
> > or at newer frameworks a tag bases approach seams more often used. 
> > 
> > In our example we use templates to map some of the parts (like the y) to
> > tags, since InfluxDB also like tags more. In this way the "prefix" could be
> > mapped to a tag.
> > 
> > But maybe we should consider to change our approach to tag-based/dimensional
> > metrics at all?!
> > 
> > In Spring Boot 2 they changed the default framework from Dropwizard to
> > Micrometer which is tag based [1]. There is a backport for Spring Boot 1.x
> > with only one added dependency - so we could just change this part.
> > 
> > Maybe we should take a look?
> > 
> > Marc
> > 
> > [1] https://spring.io/blog/2018/03/16/micrometer-spring-boot-2-s-new-applicat
> > ion-metrics-collector
> > 
> > 
> > 
> > On Wed, Jun 6, 2018 at 1:18 PM Jens Reimann <jreimann@xxxxxxxxxx> wrote:
> > > Hi everyone,
> > > 
> > > I am currently working on issue #642 [1] and have a first version with
> > > Prometheus running. Now I do have the following issue and would like to
> > > discuss how to best approach this.
> > > 
> > > I did enable metrics not only for the HTTP, MQTT and Messaging, but also
> > > for the device registry and the auth service. This allows to get metrics
> > > about the JVM (memory, threads, …) and vertx which to me would be
> > > interesting as well.
> > > 
> > > However, currently Hono does set a prefix to metrics. The default being
> > > "hono", which will be overridden by some application (e.g. "hono.mqtt").
> > > 
> > > Now this results in the same information reported in different ways. Just
> > > looking at "jvm.memory.total.used":
> > > 
> > > The Auth & Device service report it (e.g.) as:
> > > 
> > > ---
> > > hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-
> > > service-device-registry",instance="172.17.0.15:8081",job="kubernetes-
> > > service-endpoints",kubernetes_name="hono-service-device-
> > > registry",kubernetes_namespace="hono"}
> > > hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-
> > > service-auth",instance="172.17.0.17:8081",job="kubernetes-service-
> > > endpoints",kubernetes_name="hono-service-auth",kubernetes_namespace="hono"}
> > > ---
> > > 
> > > Which means that with a simple Prometheus query of
> > > "hono_jvm_memory_total_used" you can get the JVM memory usage from two
> > > applications.
> > > 
> > > If you want a specific application you need to filter:
> > > hono_jvm_memory_total_used{deploymentconfig="hono-service-auth"}
> > > 
> > > Now if you want the same information from the HTTP adapter you will need to
> > > use: hono_http_jvm_memory_heap_usage
> > > 
> > > ---
> > > hono_http_jvm_memory_heap_usage{app="hono-adapter",deploymentconfig="hono-
> > > adapter-http-vertx",instance="172.17.0.16:8081",job="kubernetes-service-
> > > endpoints",kubernetes_name="hono-adapter-http-
> > > vertx",kubernetes_namespace="hono"}
> > > ---
> > > 
> > > I do think that inconsistency should be resolved. There are two ways to do
> > > this, set a custom prefix for all Hono applications. Use the same prefix
> > > for all Hono applications.
> > > 
> > > Kubernetes itself does not use a prefix at all - query:
> > > go_memstats_alloc_bytes … results in:
> > > 
> > > ---
> > > go_memstats_alloc_bytes{instance="192.168.122.20:8443",job="kubernetes-
> > > apiservers"}
> > > go_memstats_alloc_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_
> > > os="linux",instance="localhost",job="kubernetes-
> > > nodes",kubernetes_io_hostname="localhost"}
> > > go_memstats_alloc_bytes{app="prometheus",instance="172.17.0.13:8443",job="k
> > > ubernetes-service-
> > > endpoints",kubernetes_name="prometheus",kubernetes_namespace="kube-
> > > system",name="prometheus"}
> > > go_memstats_alloc_bytes{app="openshift-web-
> > > console",instance="172.17.0.3:8443",job="kubernetes-service-
> > > endpoints",kubernetes_name="webconsole",kubernetes_namespace="openshift-
> > > web-console"}
> > > ---
> > > 
> > > There is no application specific prefix and proper tagging is used.
> > > 
> > > So my proposal would be to completely drop the prefix as I don't think it
> > > adds much and only makes things more complex. Instead using tagging of the
> > > values (like seen above).
> > > 
> > > A compromise could be, although I would like to avoid this, to make the
> > > prefix configurable, and allowing an empty prefix in the process.
> > > 
> > > Let me know what you think.
> > > 
> > > Cheers
> > > 
> > > Jens
> > > 
> > > [1] https://github.com/eclipse/hono/issues/642
> > > 
> > > -- 
> > > Jens Reimann
> > > Senior Software Engineer / EMEA ENG Middleware
> > > Werner-von-Siemens-Ring 14
> > > 85630 Grasbrunn
> > > Germany
> > > phone: +49 89 2050 71286
> > > ___________________________________________________________________________
> > > __
> > > 
> > > Red Hat GmbH, www.de.redhat.com,
> > > Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB
> > > 153243,
> > > Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
> > > Michael O'Neill
> > > _______________________________________________
> > > hono-dev mailing list
> > > hono-dev@xxxxxxxxxxx
> > > To change your delivery options, retrieve your password, or unsubscribe
> > > from this list, visit
> > > https://dev.eclipse.org/mailman/listinfo/hono-dev
> > 
> > _______________________________________________
> > hono-dev mailing list
> > hono-dev@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or unsubscribe from
> > this list, visit
> > https://dev.eclipse.org/mailman/listinfo/hono-dev
> > 
> 
> 
> 
> _______________________________________________
> hono-dev mailing list
> hono-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/hono-dev

Back to the top