Hi everyone,
I am currently working on issue #642 [1] and have a first version with Prometheus running. Now I do have the following issue and would like to discuss how to best approach this.
I did enable metrics not only for the HTTP, MQTT and Messaging, but also for the device registry and the auth service. This allows to get metrics about the JVM (memory, threads, …) and vertx which to me would be interesting as well.
However, currently Hono does set a prefix to metrics. The default being "hono", which will be overridden by some application (e.g. "hono.mqtt").
Now this results in the same information reported in different ways. Just looking at "jvm.memory.total.used":
The Auth & Device service report it (e.g.) as:
---
hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-service-device-registry",instance="
172.17.0.15:8081",job="kubernetes-service-endpoints",kubernetes_name="hono-service-device-registry",kubernetes_namespace="hono"}
hono_jvm_memory_total_used{app="hono-service",deploymentconfig="hono-service-auth",instance="
172.17.0.17:8081",job="kubernetes-service-endpoints",kubernetes_name="hono-service-auth",kubernetes_namespace="hono"}
---
Which means that with a simple Prometheus query of "hono_jvm_memory_total_used" you can get the JVM memory usage from two applications.
If you want a specific application you need to filter: hono_jvm_memory_total_used{deploymentconfig="hono-service-auth"}
Now if you want the same information from the HTTP adapter you will need to use: hono_http_jvm_memory_heap_usage
---
hono_http_jvm_memory_heap_usage{app="hono-adapter",deploymentconfig="hono-adapter-http-vertx",instance="
172.17.0.16:8081",job="kubernetes-service-endpoints",kubernetes_name="hono-adapter-http-vertx",kubernetes_namespace="hono"}
---
I do think that inconsistency should be resolved. There are two ways to do this, set a custom prefix for all Hono applications. Use the same prefix for all Hono applications.
Kubernetes itself does not use a prefix at all - query: go_memstats_alloc_bytes … results in:
---
go_memstats_alloc_bytes{instance="
192.168.122.20:8443",job="kubernetes-apiservers"}
go_memstats_alloc_bytes{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_os="linux",instance="localhost",job="kubernetes-nodes",kubernetes_io_hostname="localhost"}
go_memstats_alloc_bytes{app="prometheus",instance="
172.17.0.13:8443",job="kubernetes-service-endpoints",kubernetes_name="prometheus",kubernetes_namespace="kube-system",name="prometheus"}
go_memstats_alloc_bytes{app="openshift-web-console",instance="
172.17.0.3:8443",job="kubernetes-service-endpoints",kubernetes_name="webconsole",kubernetes_namespace="openshift-web-console"}
---
There is no application specific prefix and proper tagging is used.
So my proposal would be to completely drop the prefix as I don't think it adds much and only makes things more complex. Instead using tagging of the values (like seen above).
A compromise could be, although I would like to avoid this, to make the prefix configurable, and allowing an empty prefix in the process.
Let me know what you think.
Cheers
Jens
--
Jens Reimann
Senior Software Engineer / EMEA ENG Middleware
Werner-von-Siemens-Ring 14
85630 Grasbrunn
Germany
phone: +49 89 2050 71286
_____________________________________________________________________________
Red Hat GmbH,
www.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill
_______________________________________________