Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [hono-dev] Migration to Prometheus and metrics prefixes 
On Wed, 2018-06-06 at 15:08 +0200, Jens Reimann wrote:
> Yes. I think we should do a Spring Boot upgrade at a later time.
> 

Or we do a Spring Boot upgrade first (now that we have a good reason to do so)
and migrate to Micrometer later ...

> On Wed, Jun 6, 2018 at 3:00 PM, Hudalla Kai (INST/ECS4) <kai.hudalla@bosch-
> si.com> wrote:
> > On Wed, 2018-06-06 at 14:41 +0200, Jens Reimann wrote:
> > > Hm … I do like this approach … but it make the current task a lot bigger.
> > > However it also looks like there is better integration with all components.
> > > There is support for Prometheus out of the box and support for vertx as
> > well.
> > > 
> > > But if we want to do this in the near future, then right now might be the
> > best
> > > timing. As we need to re-do the Graphana dashboards in any case. Doing it
> > twice
> > > doesn't sound like a good idea.
> > > 
> > > So if everyone is on board with this idea, I would like give that a try. 
> > > 
> > 
> > Meaning: Using the back ported Micrometer based metrics from Spring Boot 2.0?
> > 
> > > 
> > > 
> > > On Wed, Jun 6, 2018 at 1:58 PM, Marc Pellmann <pellmann@xxxxxxxxx> wrote:
> > > > Hi Jens,
> > > > 
> > > > we use the "traditional"/hierarchical naming of the metrics like x.y.z.
> > > > Dropwizard (the used framework) typically does it this way. In newer
> > versions
> > > > or at newer frameworks a tag bases approach seams more often used. 
> > > > 
> > > > In our example we use templates to map some of the parts (like the y) to
> > > > tags, since InfluxDB also like tags more. In this way the "prefix" could
> > be
> > > > mapped to a tag.
> > > > 
> > > > But maybe we should consider to change our approach to tag-
> > based/dimensional
> > > > metrics at all?!
> > > > 
> > > > In Spring Boot 2 they changed the default framework from Dropwizard to
> > > > Micrometer which is tag based [1]. There is a backport for Spring Boot
> > 1.x
> > > > with only one added dependency - so we could just change this part.
> > > > 
> > > > Maybe we should take a look?
> > > > 
> > > > Marc
> > > > 
> > > > [1] https://spring.io/blog/2018/03/16/micrometer-spring-boot-2-s-new-appl
> > icat
> > > > ion-metrics-collector
> > > > 
> > > > 
> > > > 
> > > > On Wed, Jun 6, 2018 at 1:18 PM Jens Reimann <jreimann@xxxxxxxxxx> wrote:
> > > > > Hi everyone,
> > > > > 
> > > > > I am currently working on issue #642 [1] and have a first version with
> > > > > Prometheus running. Now I do have the following issue and would like to
> > > > > discuss how to best approach this.
> > > > > 
> > > > > I did enable metrics not only for the HTTP, MQTT and Messaging, but
> > also
> > > > > for the device registry and the auth service. This allows to get
> > metrics
> > > > > about the JVM (memory, threads, …) and vertx which to me would be
> > > > > interesting as well.
> > > > > 
> > > > > However, currently Hono does set a prefix to metrics. The default being
> > > > > "hono", which will be overridden by some application (e.g.
> > "hono.mqtt").
> > > > > 
> > > > > Now this results in the same information reported in different ways.
> > Just
> > > > > looking at "jvm.memory.total.used":
> > > > > 
> > > > > The Auth & Device service report it (e.g.) as:
> > > > > 
> > > > > ---
> > > > > hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-
> > > > > service-device-registry",instance="172.17.0.15:8081",job="kubernetes-
> > > > > service-endpoints",kubernetes_name="hono-service-device-
> > > > > registry",kubernetes_namespace="hono"}
> > > > > hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-
> > > > > service-auth",instance="172.17.0.17:8081",job="kubernetes-service-
> > > > > endpoints",kubernetes_name="hono-service-
> > auth",kubernetes_namespace="hono"}
> > > > > ---
> > > > > 
> > > > > Which means that with a simple Prometheus query of
> > > > > "hono_jvm_memory_total_used" you can get the JVM memory usage from two
> > > > > applications.
> > > > > 
> > > > > If you want a specific application you need to filter:
> > > > > hono_jvm_memory_total_used{deploymentconfig="hono-service-auth"}
> > > > > 
> > > > > Now if you want the same information from the HTTP adapter you will
> > need to
> > > > > use: hono_http_jvm_memory_heap_usage
> > > > > 
> > > > > ---
> > > > > hono_http_jvm_memory_heap_usage{app="hono-
> > adapter",deploymentconfig="hono-
> > > > > adapter-http-vertx",instance="172.17.0.16:8081",job="kubernetes-
> > service-
> > > > > endpoints",kubernetes_name="hono-adapter-http-
> > > > > vertx",kubernetes_namespace="hono"}
> > > > > ---
> > > > > 
> > > > > I do think that inconsistency should be resolved. There are two ways to
> > do
> > > > > this, set a custom prefix for all Hono applications. Use the same
> > prefix
> > > > > for all Hono applications.
> > > > > 
> > > > > Kubernetes itself does not use a prefix at all - query:
> > > > > go_memstats_alloc_bytes … results in:
> > > > > 
> > > > > ---
> > > > > go_memstats_alloc_bytes{instance="192.168.122.20:8443",job="kubernetes-
> > > > > apiservers"}
> > > > >
> > go_memstats_alloc_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_
> > > > > os="linux",instance="localhost",job="kubernetes-
> > > > > nodes",kubernetes_io_hostname="localhost"}
> > > > >
> > go_memstats_alloc_bytes{app="prometheus",instance="172.17.0.13:8443",job="k
> > > > > ubernetes-service-
> > > > > endpoints",kubernetes_name="prometheus",kubernetes_namespace="kube-
> > > > > system",name="prometheus"}
> > > > > go_memstats_alloc_bytes{app="openshift-web-
> > > > > console",instance="172.17.0.3:8443",job="kubernetes-service-
> > > > >
> > endpoints",kubernetes_name="webconsole",kubernetes_namespace="openshift-
> > > > > web-console"}
> > > > > ---
> > > > > 
> > > > > There is no application specific prefix and proper tagging is used.
> > > > > 
> > > > > So my proposal would be to completely drop the prefix as I don't think
> > it
> > > > > adds much and only makes things more complex. Instead using tagging of
> > the
> > > > > values (like seen above).
> > > > > 
> > > > > A compromise could be, although I would like to avoid this, to make the
> > > > > prefix configurable, and allowing an empty prefix in the process.
> > > > > 
> > > > > Let me know what you think.
> > > > > 
> > > > > Cheers
> > > > > 
> > > > > Jens
> > > > > 
> > > > > [1] https://github.com/eclipse/hono/issues/642
> > > > > 
> > > > > -- 
> > > > > Jens Reimann
> > > > > Senior Software Engineer / EMEA ENG Middleware
> > > > > Werner-von-Siemens-Ring 14
> > > > > 85630 Grasbrunn
> > > > > Germany
> > > > > phone: +49 89 2050 71286
> > > > >
> > ___________________________________________________________________________
> > > > > __
> > > > > 
> > > > > Red Hat GmbH, www.de.redhat.com,
> > > > > Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen,
> > HRB
> > > > > 153243,
> > > > > Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham,
> > > > > Michael O'Neill
> > > > > _______________________________________________
> > > > > hono-dev mailing list
> > > > > hono-dev@xxxxxxxxxxx
> > > > > To change your delivery options, retrieve your password, or unsubscribe
> > > > > from this list, visit
> > > > > https://dev.eclipse.org/mailman/listinfo/hono-dev
> > > > 
> > > > _______________________________________________
> > > > hono-dev mailing list
> > > > hono-dev@xxxxxxxxxxx
> > > > To change your delivery options, retrieve your password, or unsubscribe
> > from
> > > > this list, visit
> > > > https://dev.eclipse.org/mailman/listinfo/hono-dev
> > > > 
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > hono-dev mailing list
> > > hono-dev@xxxxxxxxxxx
> > > To change your delivery options, retrieve your password, or unsubscribe
> > from
> > > this list, visit
> > > https://dev.eclipse.org/mailman/listinfo/hono-dev
> > _______________________________________________
> > hono-dev mailing list
> > hono-dev@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or unsubscribe from
> > this list, visit
> > https://dev.eclipse.org/mailman/listinfo/hono-dev
> 
> 
> 
> _______________________________________________
> hono-dev mailing list
> hono-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/hono-dev

Back to the top