Home » Archived » Eclipse SmartHome » New Prometheus Metrics bundle - Need help attaching to Pax logging
| New Prometheus Metrics bundle - Need help attaching to Pax logging [message #1792637] |
Fri, 20 July 2018 11:50  |
 Hanno Wagner
Messages: 12
Registered: April 2017 |
Junior Member |
|
|
Hello everyone,
we are developing an add-on bundle for Eclipse Smart Home/OpenHAB, experience some difficulties with it and are looking for help here now. We tried out a lot of different approaches but all failed. I think, the bundle will be helpful for others too, so we release it under EPL, and maybe it is interesting enough to even directly incorporate it into upstream later.
The bundle exposes internal metrics of ESH/openHAB in Prometheus' format (https://prometheus.io/docs/concepts/data_model/) so it can be scraped/ingested there and fed into a Grafana board, to get a good understanding of the system's health, potentially especially for multiple systems deployed in the field, not directly physically accessible to the one who monitors it.
The metrics there currently are
- openhab_bundle_state{bundle="[BUNDLENAME]"} [BUNDLESTATE]
- openhab_thing_state{thing="[THINGID]"} [THINGSTATE]
- openhab_inbox_count [INBOXCOUNT]
- smarthome_event_count{source="[BUNDLENAME]"} [EVENTCOUNT]
We now would like to add internal states of several bindings (most importantly Z-wave), which are hard to aquire, because they are internal. But regularly, the bindings output quite a lot of interesting information (e.g. size of send queue, number of sending attempts) to the logs in DEBUG level.
It will, in the general approach, add more metrics to the output like
- openhab_logmessages_total{level="[LOGLEVEL]"} [MESSAGECOUNT]
- openhab_logmessages_error{type="[LOGGERNAME]"} [MESSAGECOUNT]
Afterwards we probably want to filter, process and parse a custom set of messages to get e.g. Z-waves internal message queque etc.
The Prometheus javaclient library which we use and which comes as OSGi bundles, has logging collectors for log4j, log4j2 and logback (https://github.com/prometheus/client_java#logging), so the "only" thing we'd need to do is to setup our bundle (or a class in it) as an appender to the openHAB logging facility, to get the information parallel to the usual file logging.
And here comes the problem: We were not able to figure out how to hook it up to the Karaf/Felix/Pax in the distribution runtime (we managed to do it for Logback/Equinox inside the development IDE).
Can anybody here point us in the right direction, how or where we need to register our bundle/fragment so that all the logging messages can be configured to end up here? We googled a lot, but it seems nobody had that exact same requirement earlier, all articles and hints we found are outdated or only providing bits or are based on different requirements.
You can see the current state here: https://github.com/KuguHome/openhab-prometheus-metrics , build instructions etc. are included in the README.md, a working jar is here: https://github.com/KuguHome/openhab-prometheus-metrics/releases/download/v0.9/com.kuguhome.openhab.prometheusmetrics-2.4.0-SNAPSHOT.1.jar. To make it run in the `addons/` folder, you will need to add two more files:
The master branch has the working version, which is missing the `openhab_logmessages_` metrics, the `eclipse-workable-logback-metrics` branch has those metrics, but will only work in the Eclipse IDE, not inside the standalone distribution.
An example scrape looks like this: https://github.com/KuguHome/openhab-prometheus-metrics/wiki/Example-scrape
We have set it up on a couple of running instances and this is how it looks like when scraped by Prometheus and visualized by Grafana:  https://user-images.githubusercontent.com/3128983/42999260-5d1de3ea-8c1d-11e8-9713-1c9bc8e43ef6.png
The Grafana Dashboard code is here: https://github.com/KuguHome/openhab-prometheus-metrics/wiki/Grafana-source
What we did until now to get it running inside openHAB distribution runtime:
- added appender to org.ops4j.pax.logging.cfg
- put jar bundle into maven repo according to org.ops4j.pax.url.mvn.cfg
(so bundle should be picked up at startup).
- put line of appender here startup.properties
mvn:io.prometheus/simpleclient_log4j2/0.4.0 = 8
mvn:org.ops4j.pax.logging/pax-logging-api/1.10.1 = 8 <---- already was there
Got that:
2018-07-04 17:34:59,960 CM Configuration Updater (Update: pid=org.ops4j.pax.logging) ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.config.AppendersPlugin for element Appenders. java.lang.NullPointerException
Please let me know if you need any more information.
Also we are very interested in your opinion of the bundle or any other kind of feedback.
Some more insights about our process of thoughts:
- We don't want to read the Log from the files, because it seems very ineffcient to parse things which were there as objects before and then have been serialized/formattet
- We don't want to use a Syslog output and pipe it into a solution like Logstash/Elasticsearch etc. because that's as inefficient (object->format->parse) and needs much more setup overhead, other things running on the openHAB machine
- We looked into the Openhab Log Viewer (which is running independently on a different Port 9001) and the Log Reader Binding, but they do the same (read the files) and are also not helping our target of getting internal states onto a time-based, visually appealing graph for good diagnostics
- Rather, we want to hook into the stream of raw object data directly inside of openHAB, using the APIs that are there in Logback/Log4j(2)/Pax for this exact purpose to build an own collector
- There was some talk about Promethes style metrics earlier here: https://community.openhab.org/t/overview-of-expectable-command-latency-on-different-hardware/5968/11 but we think the Hardware/OS data (memory, CPU etc.) is easier to aquire by running a node_exporter next to openHAB and scrape it from there.
Looking forward to your responses,
Cheers,
Hanno - Felix Wagner (Product Manager for Kugu Home GmbH)
PS: I crosspost this to the openHAB forum, because it might be interesting to users there (https://community.openhab.org/t/new-prometheus-metrics-bundle-need-help-for-next-step/48094)
|
|
| | | | |
| Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1793459 is a reply to message #1793426] |
Wed, 08 August 2018 16:55  |
Jochen Hiller
Messages: 11
Registered: July 2009 |
Junior Member |
|
|
Hi Hanno,
thanks for your contribution and ideas about monitoring.
We (Deutsche Telekom) are also using Eclipse SmartHome in our commercial product and have existing solutions about monitoring (incl. metrics) in place, where I would like to share some insights.
We decided for gathering metrics NOT to rely on logging information, as we want to get much more details about the gateway where the software stack is running. From Linux OS, from JVM, from our application stack.
We decided to use the Eclipse Microprofile Metrics API (see http://microprofile.io/project/eclipse/microprofile-metrics) as the API we support inside our stack. OSGi bundles from the platform, from bindings, from partner applications can contribute (=register) metrics. We have a (more simple) implementation just querying the metrics from a servlet which register itself at /metrics and provide data in Prometheus format.
We decided NOT to use DropWizard, as we would like to stick to a "standard" and Eclipse APIs where possible, and avoid external dependencies where possible. We had to reduce the API a little bit as it uses some CDI annotations (javax.*) we do not support on Java8 compact 2 profile.
In addition to local scraping by Prometheus we can also (if allowed due to privacy regulations) push the data to Prometheus Push Gateway where we collect the data while Prometheus in the cloud can scrape the Push Gateway on a regular base. We use also Grafana as frontend on Prometheus.
Actually we collect these kind of metrics:
* core OS (/proc, /sysfs filesystems): CPU, IO, memory
* JVM metrics (using JMX on compact 3 where available)
* log messages counting (using OSGi LogService)
* application specific metrics
We do not add bundle states, as we have our own management server in place, so no need for it.
So the approach is slightly different by adding a lightweight implementation, compact 2 compliant, no external dependencies, and not rely on the logging infrastructure.
For health checks we have a proprietary solution in place, where we also think to migrate to Eclipse Microprofile Health API (http://microprofile.io/project/eclipse/microprofile-health) in future. I would be interested to understand your requirements and ideas here too.
I think we could contribute some parts of our solution to Eclipse SmartHome if there is broader interest in that.
What do you think?
Bye, Jochen
I am unfortunately not at FrOSCon, but will join the EclipseCon Europe in October. Otherwise we can also align on other channels.
|
|
| |
| Re: [SOLVED] New Prometheus Metrics bundle - Writing own PaxAppender for Karaf [message #1794200 is a reply to message #1793459] |
Mon, 27 August 2018 12:22 |
Hanno Wagner
Messages: 12
Registered: April 2017 |
Junior Member |
|
|
Hi Jochen,
thank you for your insightful answer and sorry for the late replay. I just overlooked the post because I received no notification :-(
Much of what you are writing sounds to be very compatible to our approach.
We also don't want to rely on the logging but rather get the information directly from where it is already available in the code. But in a step-by-sted way, we thought it might be quickest, to get some metrics by parsing the log entries first and later shift to direct sourcing.
I hadn't seen the Eclipse Microprofile Metrics API before, but that looks indeed very promising. The way it currently works for us is also pretty similar to what you describe:
Our bundle provides a /rest/metrics/prometheus endpoint, which outputs the metrics. They are collected in a modular way via OSGi components, maybe see here https://github.com/KuguHome/openhab-prometheus-metrics/blob/master/com.kuguhome.openhab.prometheusmetrics/src/main/java/com/kuguhome/openhab/prometheusmetrics/exposable/InboxCountMetric.java for an example. So this way, binding developers could quite easily add their own metrics to expose. We also wanted to further simplify the API to have something like Metrics.setMetric("name",value); for very quick usage.
So if we can combine our efforts, we should be able to get this into ESH mainline in a reasonable time. If this suits you, I would like to discuss it with you in a video chat? Just let me know when you are available again (h.wagner@kugu-home.com).
Cheers,
Hanno
|
|
| | |
Goto Forum:
Current Time: Thu Dec 26 21:13:31 GMT 2024
Powered by FUDForum. Page generated in 0.04955 seconds |
] 
Search
Help
Register
Login
Home
