On Linux, we get more detailed interface statistics from netlink than we did
from `/proc/net/dev`.
This commit adds a new flag (`--collector.netdev.enable-detailed-metrics`) to
expose those statistics under new (incompatible) metric names. When enabled,
the metric names are also changed on Darwin and BSD platforms to keep
everything consistent, but it doesn't provide more detailed statistics on those
platforms.
The old metrics can be derived from the new ones using the following rules
([dev_seq_printf_stats]):
- `receive_errs` = `receive_errors`
- `receive_drop` = `receive_dropped` + `receive_missed_errors`
- `receive_fifo` = `receive_fifo_errors`
- `receive_frame` = `receive_length_errors` + `receive_over_errors` + `receive_crc_errors` + `receive_frame_errors`
- `receive_multicast` = `multicast`
- `transmit_errs` = `transmit_errors`
- `transmit_drop` = `transmit_dropped`
- `transmit_fifo` = `transmit_fifo_errors`
- `transmit_colls` = `collisions`
- `transmit_carrier` = `transmit_aborted_errors` + `transmit_carrier_errors` + `transmit_heartbeat_errors` + `transmit_window_errors`
[dev_seq_printf_stats]: https://github.com/torvalds/linux/blob/master/net/core/net-procfs.c#L75-L97
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
To prepare for the introduction of new metric names, add tests for the legacy
metric names and values. This will make it easier to ensure that the code that
converts the new metrics to the old ones (for compatibility) behaves correctly.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
Since netdev metrics are now read from netlink instead of `/proc/net/dev`, we
can't easily spoof them for the end-to-end tests by reading a fixture file in
place of `/proc/net/dev`.
Therefore, we only get metrics for `lo` and ignore those that would return
unpredictable values (i.e. the byte and packet counters).
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
Instead of parsing `/proc/net/dev` to get network interface statistics, get
them from a netlink call.
Internally, both come from the [rtnl_link_stats64] struct, but with
`/proc/net/dev`, some of the values are aggregated together in
[dev_seq_printf_stats], so we get less information out of them.
This commit maintains compatibility by aggregating those stats back into the
same metrics.
[rtnl_link_stats64]: https://github.com/torvalds/linux/blob/master/include/uapi/linux/if_link.h#L42-L246
[dev_seq_printf_stats]: https://github.com/torvalds/linux/blob/master/net/core/net-procfs.c#L75-L97
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
These two memory classes have been here for a while now in FreeBSD,
adding them allows having information for all memory classes.
Signed-off-by: François Charlier <fcharlier@ploup.net>
Log a single error message when the udev data directory (`/run/udev/data` by
default) is unreadable, and then don't try to get device properties out of it.
Also lower the log level from error to debug when we can't parse the udev files
properly, since these messages would be sent every time the node exporter gets
scraped.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
When parsing udev data, skip lines that don't start with `E:`.
Lines prefixed with `E:` represent device properties, as documented in
udevadm(8).
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
Set the `--path.udev.data` flag to point to the udev fixture, and update the
output fixture with
```console
$ ./end-to-end-test.sh -u
```
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
Now that we read some data from `/run/udev/data`, add the corresponding
fixtures and update the expected test results accordingly.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
Instead of hard-coding the path to `/run/udev/data`, intoduce a
`--path.udev.data` flag that defaults to that value.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
Add labels to the `node_disk_info` metric extracted from udev, such as `model`,
`path`, `revision`, `serial` and `wwn`.
Also add a few metrics related to filesystem and device mapper, which are also
extracted from udev information.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
Use standard include/exclude pattern for device include/exclude in the
diskstats collector.
Signed-off-by: Ben Kochie <superq@gmail.com>
Co-authored-by: rushilenekar20 <rushilenekar20@gmail.com>
Fix up handling of CPU info collector on non-x86_64 systems due to
fixtures containing `/proc/cpuinfo` from x86_64.
* Update e2e 64k page test fixture from an arm64 system.
* Enable ARM testing in CircleCI.
Fixes: https://github.com/prometheus/node_exporter/issues/1959
Signed-off-by: Ben Kochie <superq@gmail.com>
* Correctly name collector file.
* Fix cgroup summary type as gauge.
* Use a boolean metric rather than a label for enabled.
Signed-off-by: Ben Kochie <superq@gmail.com>
Use unix.ByteSliceToString to convert Utsname []byte fields to strings.
This also allows to drop the bytesToString helper which serves the same
purpose and matches ByteSliceToString's implementation.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
This is necessary to build on darwin using nix, as nix-darwin uses an
older macOS SDK, built from Apple's open source releases.
Signed-off-by: Peter Woodman <peter@shortbus.org>
In certain instances on heavily loaded nodes with many network
devices, there may be concurrent access to the netdev collector's
`metricDescs` map, resulting in a panic. This adds a mutex to prevent
concurrent reads and writes to the map.
Signed-off-by: Brad Ison <bison@xvdf.io>
Move the systemd version function to an exporter method. This way we can
update the Verison information at every scrape, in case the underlying
version changes.
Signed-off-by: Ben Kochie <superq@gmail.com>
systemd patch versions are as important as the major version number;
they indicate security or bug fixes or other behavioural changes between
versions.
Use float64 over float32 as the rounding error with float32 rendered
250.3 as 250.3000030517578 in my testing.
Signed-off-by: Joe Groocock <jgroocock@cloudflare.com>
Signed-off-by: Joe Groocock <me@frebib.net>
analogous to the /var/lib/docker exclude added in
https://github.com/prometheus/node_exporter/pull/814
podman rootful containers mount eg. shm filesystems at
/var/lib/containers/storage/*-containers/*/userdata/shm. these should be
treated like things under /var/lib/docker by default.
Signed-off-by: Lauri Tirkkonen <lauri@hacktheplanet.fi>
Allow filtering APR entries based on device. Useful for ignoring
entries for network namespaces (containers).
Signed-off-by: Ben Kochie <superq@gmail.com>
This adds a new Linux metric, node_softirqs_total, which corresponds
to the 'softirq' line in /proc/stat. This metric is disabled by
default and it can be enabled with '--collector.stat.softirq'.
Signed-off-by: Jacob Vosmaer <jacob@gitlab.com>
Use the non-cgo version for all openbsd architectures.
The old code only pulled some defines from header files. Just add them
as enumerations in native go. Also be careful at what the SysctlRaw returns.
Implement a way that supports both recent and old pre-6.4 OpenBSD systems.
With go-1.16 OpenBSD binaries will link to libc and because of this binaries
built on OpenBSD 6.9-current do not run on OpenBSD 6.3. OpenBSD 6.3 is also
not supported for more then 2 years. So maybe the compat code is not needed.
Still validation object length before doing an unsafe pointer conversion
is probably reasonable but I'm no golang expert.
Signed-off-by: Claudio Jeker <claudio@openbsd.org>
TCP timeouts count is a useful signal to show
abnormal network performance and is another
signal to aid debugging. This metric can be
used to generate proactive alerts for host
network namespace workloads.
Signed-off-by: Martin Kennelly <mkennell@redhat.com>
The new `lnstat` collector produces a high number of metrics, per-cpu,
and results in approximately double the number of metrics previously
scraped. For example, a typical server with 64 cores produces 3832
lnstat metrics compared to 4147 metrics for the remaining collectors.
Therefore disable the `lnstat` collector by default.
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
Sanitizing the metric names can lead to duplicate metric names:
```
caller=level.go:63 level=error caller="error gathering metrics: [from Gatherer #2] collected metric \"node_ethtool_giant_hdr\" { label:<name:\"device\" value:\"ens192\" > untyped:<value:0" msg=" > } was collected before with the same name and label values"
```
Generate a map from the sanitized metric names to the metric names from
ethtool. In case of duplicate sanitized metric names drop both metrics,
because it is unknown which one to take.
Fixes: https://github.com/prometheus/node_exporter/issues/2185
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
Use SysctlTimeval from the golang.org/x/sys/unix package to
simplify the implementation of the boottime collector for the BSDs and
allows to build it without cgo.
Tested on macOS 11.6, FreeBSD 13 and OpenBSD 7.
Signed-off-by: Tobias Klauser <tklauser@distanz.ch>
Add a DMI collector to expose the Desktop Management Interface (DMI)
info from `/sys/class/dmi/id/`. This will expose information about the
BIOS, mainboard, chassis, and product.
Closes: https://github.com/prometheus/node_exporter/issues/303
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
Use `time.NewTimer()` and explicit `Stop()` to avoid memory bloat / GC problems with `time.After()` in the Linux filesystem collector timeout handling.
Signed-off-by: bawenmao <bawenmao@sogou-inc.com>
The ethtool_cmd struct from the linux kernel contains information about the speeds and features supported by a
network device. This includes speeds and duplex but also features like autonegotiate and 802.3x pause frames.
Closes#1444
Signed-off-by: W. Andrew Denton <git@flying-snail.net>
* collector: Unwrap glob textfile directories
* collector: Store full path in mtime's file label
The point is to avoid duplicated gauges from files with the same name in
different directories.
This introduces support for exporting from multiple directories matching
given pattern (e.g. `/home/*/metrics/`).
Signed-off-by: Kiril Vladimirov <kiril@vladimiroff.org>
Expose GPU metrics using `sysfs/drm`.
`amdgpu` is the only driver which exposes this information through DRM.
Signed-off-by: Siavash Safi <siavash.safi@gmail.com>
Use the same flag pattern as netdev to make filtering methods the same.
* Move SanitizeMetricName to helper.go
Signed-off-by: Ben Kochie <superq@gmail.com>
* Refactor diskstats_linux to use procfs.
* Add `node_disk_info` metric.
Signed-off-by: W. Andrew Denton <git@flying-snail.net>
Co-authored-by: W. Andrew Denton <git@flying-snail.net>
Currently Node Exporter has a metric called `node_uname_info` which of
course exposes uname info. While this is nice, it does not help if you
are running different OSes which could have similar uname info.
Therefore parse `/etc/os-release` or `/usr/lib/os-release` and expose a
`node_os_info` metric which provide information regarding the OS
release/version of the node. Also expose the major.minor part of the OS
release version as `node_os_version`.
Since the os-release files will not change often, cache the parsed
content and only refresh the cache if the modification time changes.
This `os` collector will read files outside of `/proc` and `/sys`, but
the os-release file is widely used and the format is standardized:
https://www.freedesktop.org/software/systemd/man/os-release.html
Bug: https://github.com/prometheus/node_exporter/issues/1574
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
In high scale virtualized / cloud environments there are typically
no guest VMs. Add a boolean flag to allow disabling the Linux guest
CPU metrics.
Signed-off-by: Ben Kochie <superq@gmail.com>
Add a `node_ethtool_info` metric to all ethtool devices to expose driver
information with following labels:
* bus_info
* driver
* expansion_rom_version
* firmware_version
* version
This metric is useful to monitor the firmware version to be up-to-date.
Note: The version label might be malformed due to bug #39 in ethtool:
https://github.com/safchain/ethtool/issues/39
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
OpenMetrics and the Prometheus exposition format require the metric name
to consist only of alphanumericals and "_", ":" and they must not start
with digits. The metric names from the ethtool stats might contain
spaces, brackets, and dots. Converting them directly to metric names
will produce invalid metric names.
Therefore sanitize the metric names and convert them to lower case.
Fixes: https://github.com/prometheus/node_exporter/issues/2083
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
This adds a new flag --collector.ethtool.metrics-include to the ethtool
collector. Only metrics matching this regexp will be collected.
Signed-off-by: Johannes 'fish' Ziemke <github@freigeist.org>
Other network related collectors allow to filter out unwanted devices.
Add this support to the new ethtool collector as well.
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
Update procfs library to include ignored fields ParseInt handling.
Wrap error returns so that the user can know more about what failed.
Returns from getAllocatedThreads() are errors anyway.
Fixes: https://github.com/prometheus/node_exporter/issues/2110
Signed-off-by: Ben Kochie <superq@gmail.com>
The Linux CPU idle stat can also jump backwards slightly in some cases.
Allow the jump back up to 3 seconds before we attempt to reset the CPU
counter cache.
Fixes: https://github.com/prometheus/node_exporter/issues/1903
Signed-off-by: Ben Kochie <superq@gmail.com>
Add a collector for NVMes to expose the firmware versions. This requires
procfs >= 0.7.0.
Fixes#1891
Signed-off-by: Benjamin Drung <benjamin.drung@ionos.com>
We should filter excluded interfaces before parsing the interface
details.
This change is based on https://github.com/prometheus/procfs/pull/376
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>
When /metrics is called for specific collectors, the collectors are
initialed every time. Which means that we spend a lot of time
re-initiating the same collectors again and again. Especially, some
collectors make the assumptions that that are initiated once - e.g.
systemd collector generates info message upon initiation.
Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>
Fix the error logging of the promhttp handler by connecting it to the
promlog setup.
* Switch to go-kit/log.
* Cleanup CHANGELOG.
Fixes: https://github.com/prometheus/node_exporter/issues/1886
Signed-off-by: Ben Kochie <superq@gmail.com>
Add darwin/arm64 to the CGO crossbuilder list.
* Update Makefile.common to pick up new promu.
* Fix possible nil pointer caught by staticcheck.
* Update collector build tags.
https://github.com/prometheus/node_exporter/issues/1997
Signed-off-by: Ben Kochie <superq@gmail.com>
Some devices (ex virtual) don't have a speed and report `-1` as the
speed value. Add a flag to allow ignoring speed on these devices.
Fixes: https://github.com/prometheus/node_exporter/issues/1967
Signed-off-by: Ben Kochie <superq@gmail.com>
Avoid panic on invalid UTF-8 from /sys/class/power_supply by
sanitizing strings parsed from the kernel.
* Add a broken string to the test fixtures.
Fixes: https://github.com/prometheus/node_exporter/issues/1979
Signed-off-by: Ben Kochie <superq@gmail.com>
When CONFIG_PSI_DEFAULT_DISABLED=y, the pressure system returns
"operation not supported", rather than permission denied or not
exposing the /proc/pressure files.
Fixes: https://github.com/prometheus/node_exporter/issues/1961
Signed-off-by: Ben Kochie <superq@gmail.com>
* Update Build
- Update CircleCI orb.
- Update CIrcleCI Machine image.
- Use golang-builder 1.15.
* Update Go modules.
* Fixup fixtures for XFS bug.
NOTE: We have improved some of the flag naming conventions (PR #1743). The old names are
deprecated and will be removed in 2.0. They will continue to work for backwards
compatibility.
* [CHANGE] Improve filter flag names #1743
* [CHANGE] Add btrfs and powersupplyclass to list of exporters enabled by default #1897
* [FEATURE] Add fibre channel collector #1786
* [FEATURE] Expose cpu bugs and flags as info metrics. #1788
* [FEATURE] Add network_route collector #1811
* [FEATURE] Add zoneinfo collector #1922
* [ENHANCEMENT] Add more InfiniBand counters #1694
* [ENHANCEMENT] Add flag to aggr ipvs metrics to avoid high cardinality metrics #1709
* [ENHANCEMENT] Adding backlog/current queue length to qdisc collector #1732
* [ENHANCEMENT] Include TCP OutRsts in netstat metrics #1733
* [ENHANCEMENT] Add pool size to entropy collector #1753
* [ENHANCEMENT] Remove CGO dependencies for OpenBSD amd64 #1774
* [ENHANCEMENT] bcache: add writeback_rate_debug stats #1658
* [ENHANCEMENT] Add check state for mdadm arrays via node_md_state metric #1810
* [ENHANCEMENT] Expose XFS inode statistics #1870
* [ENHANCEMENT] Expose zfs zpool state #1878
* [ENHANCEMENT] Added an ability to pass collector.supervisord.url via SUPERVISORD_URL environment variable #1947
* [BUGFIX] filesystem_freebsd: Fix label values #1728
* [BUGFIX] Fix various procfs parsing errors #1735
* [BUGFIX] Handle no data from powersupplyclass #1747
* [BUGFIX] udp_queues_linux.go: change upd to udp in two error strings #1769
* [BUGFIX] Fix node_scrape_collector_success behaviour #1816
* [BUGFIX] Fix NodeRAIDDegraded to not use a string rule expressions #1827
* [BUGFIX] Fix node_md_disks state label from fail to failed #1862
* [BUGFIX] Handle EPERM for syscall in timex collector #1938
* [BUGFIX] bcache: fix typo in a metric name #1943
* [BUGFIX] Fix XFS read/write stats (https://github.com/prometheus/procfs/pull/343)
Signed-off-by: Ben Kochie <superq@gmail.com>
* Use `device` label to match other `node_network_...` metics.
* Fix naming convention to match Promehteus best practices.
Signed-off-by: Ben Kochie <superq@gmail.com>
I have rewritten all CGO dependencies for OpenBSD amd64
using pure go, be able to crosscompile node_exporter.
Signed-off-by: ston1th <ston1th@giftfish.de>
this change fixes the logging message for the filesystem ignored-fs-types flag to output the flag instead of the mountpoints flag.
Signed-off-by: xinau <felix.ehrenpfort@protonmail.com>
We've gathered enough evidence that the CPU counter bug workaround is
working as intended. Downgrade the message from Warning to Debug.
Signed-off-by: Ben Kochie <superq@gmail.com>
* Expose metric for state=check for node_md_state
* Added new e2e output fixture including md201 which is in checking state and a the new state=check labeled metric for all other md
Signed-off-by: Christian Rohmann <github@frittentheke.de>
* Check interface name before loading interface data
* Reduce indentation
* Optimize nested netdev map
This change avoids conversion to strings and back.
Signed-off-by: Julian Kornberger <jk+github@digineo.de>
