SapMachine, a downstream distribution of OpenJDK, has introduced a new monitoring tool, SapMachine Vitals, that keeps a condensed history of the operating system and JVM statistics. The monitoring feature is activated by default and may be used to retrieve information such as heap usage, metaspace size, container memory statistics and limits, the number of classes loaded and the number of threads spawned.
Vitals is a combination of sysstat operating system tools and Java Virtual Machine (JVM) statistics. The data is retrieved every ten seconds, by default, and buffered for ten days inside the JVM. The buffer contains more detailed information about the last hour. While the overhead of Vitals is relatively low, with about 300 Kb of memory consumption and little CPU consumption, it's still possible to disable the feature with the -XX:-EnableVitals command.
The jcmd <pid> VM.vitals command may be used to create the report for a running application. The report is also part of the hs_err_pid.log, which the JVM generates as part of an abnormal exit. Lastly, when the JVM is started with the -XX:+PrintVitalsAtExit argument, the Vitals report is printed to stdout before the exit is completed. Alternatively, the -XX:+DumpVitalsAtExit argument may be used to create the following files: sapmachine_vitals_pid.txt and sapmachine_vitals_pid.csv. The names of the files may be customized with the argument -XX:VitalsFile=<custom-filename>.
The various suboptions of VM.vitals are displayed with the help argument:
jcmd 11818 help VM.vitals
…
Options: (options must be specified using the <key> or <key>=<value> syntax)
scale : [optional] Memory usage in which to scale. Valid values are: k, m, g
(fixed scale) or "dynamic" for a dynamically chosen scale. (STRING, dynamic)
csv : [optional] csv format. (BOOLEAN, false)
no-legend : [optional] Omit legend. (BOOLEAN, false)
reverse : [optional] Reverse printing order. (BOOLEAN, false)
raw : [optional] Print raw values. (BOOLEAN, false)
now : [optional] Sample now values (BOOLEAN, false)
The Vitals report has three subsections: system including container stats, process stats and JVM stats. Each subsection contains a list of the metrics and their abbreviations and at the end of the report each metric is listed in a column and the measurements are displayed in the rows. In order to improve readability, we combine the metrics and measurements in this article per subsection from the jcmd 11818 VM.vitals command.
First, the list of system and container metrics and their corresponding abbreviations is displayed:
------------system------------
avail: Memory available without swapping [host] [krn]
comm: Committed memory [host]
crt: Committed-to-Commit-Limit ratio (percent) [host]
swap: Swap space used [host]
si: Number of pages swapped in [host] [delta]
so: Number of pages pages swapped out [host] [delta]
p: Number of processes
t: Number of threads
tr: Number of threads running
tb: Number of threads blocked on disk IO
cpu-us: CPU user time [host]
cpu-sy: CPU system time [host]
cpu-id: CPU idle time [host]
cpu-st: CPU time stolen [host]
cpu-gu: CPU time spent on guest [host]
cgroup-lim: cgroup memory limit [cgrp]
cgroup-slim: cgroup memory soft limit [cgrp]
cgroup-usg: cgroup memory usage [cgrp]
cgroup-kusg: cgroup kernel memory usage (cgroup v1 only) [cgrp]
The measurements related to the system and container metrics:
---------------------------------system----------------------------------
-----cpu------ -----cgroup------
avail comm crt swap si so p t tr tb us sy id st gu lim slim usg kusg
2023-02-14 19:51:57 25,7g 11,4g 64 0k 0 0 305 804 1 0 3 1 95 0 0 0k 64m
2023-02-14 19:51:47 25,7g 11,4g 64 0k 0 0 304 802 5 0 5 1 94 0 0 0k 64m
2023-02-14 19:51:37 25,7g 11,3g 64 0k 0 0 304 800 2 0 5 2 92 0 0 0k 64m
2023-02-14 19:51:27 25,7g 11,4g 64 0k 0 0 305 800 4 0 5 2 92 0 0 0k 64m
2023-02-14 19:51:17 25,7g 11,4g 64 0k 0 0 305 804 2 0 9 2 88 0 0 0k 64m
2023-02-14 19:51:07 23,8g 13,4g 76 0k 308 915 2 0 0k 26m
The list of process metrics and their corresponding abbreviations is displayed:
-----------process------------
virt: Virtual size
rss-all: Resident set size, total
rss-anon: Resident set size, anonymous memory [krn]
rss-file: Resident set size, file mappings [krn]
rss-shm: Resident set size, shared memory [krn]
swdo: Memory swapped out
cheap-usd: C-Heap, in-use allocations (may be unavailable if RSS > 4G) [glibc]
cheap-free: C-Heap, bytes in free blocks (may be unavailable if RSS > 4G) [glibc]
cpu-us: Process cpu user time
cpu-sy: Process cpu system time
io-of: Number of open files
io-rd: IO bytes read from storage or cache
io-wr: IO bytes written
thr: Number of native threads
The measurements related to the process metrics:
--------------------------process--------------------------
-------rss------- -cheap-- -cpu- ----io-----
virt all anon file shm swdo usd free us sy of rd wr thr
10,9g 85m 50m 35m 0k 0k 24m 9m 0 0 6 76k 0k 20
10,9g 85m 50m 35m 0k 0k 24m 9m 0 0 6 76k 0k 20
10,9g 85m 50m 35m 0k 0k 24m 9m 0 0 6 76k 0k 20
10,9g 85m 50m 35m 0k 0k 24m 9m 0 0 6 76k 0k 20
10,9g 85m 50m 35m 0k 0k 24m 9m 1 0 6 724k <1k 20
10,6g 36m 12m 24m 0k 0k 21m 2m 5 18
The list of JVM metrics and their corresponding abbreviations is displayed:
-------------jvm--------------
heap-comm: Java Heap Size, committed
heap-used: Java Heap Size, used
meta-comm: Meta Space Size (class+nonclass), committed
meta-used: Meta Space Size (class+nonclass), used
meta-csc: Class Space Size, committed [cs]
meta-csu: Class Space Size, used [cs]
meta-gctr: GC threshold
code: Code cache, committed
nmt-mlc: Memory malloced by hotspot [nmt]
nmt-map: Memory mapped by hotspot [nmt]
nmt-gc: NMT "gc" (GC-overhead, malloc and mmap) [nmt]
nmt-oth: NMT "other" (typically DBB or Unsafe.allocateMemory) [nmt]
nmt-ovh: NMT overhead [nmt]
jthr-num: Number of java threads
jthr-nd: Number of non-demon java threads
jthr-cr: Threads created [delta]
jthr-st: Total reserved size of java thread stacks [nmt] [linux]
cldg-num: Classloader Data
cldg-anon: Anonymous CLD
cls-num: Classes (instance + array)
cls-ld: Class loaded [delta]
cls-uld: Classes unloaded [delta]
The measurements related to the JVM metrics:
----------------------------------jvm----------------------------------
--heap--- ---------meta---------- --jthr--- --cldg-- -----cls-----
comm used comm used csc csu gctr code num nd cr num anon num ld uld
508m 23m 9m 9m 1m 1m 21m 7m 11 1 0 36 32 2515 0 0
508m 23m 9m 9m 1m 1m 21m 7m 11 1 0 36 32 2515 0 0
508m 23m 9m 9m 1m 1m 21m 7m 11 1 0 36 32 2515 0 0
508m 23m 9m 9m 1m 1m 21m 7m 11 1 0 36 32 2515 0 0
508m 23m 9m 9m 1m 1m 21m 7m 11 1 2 36 32 2515 1841 0
508m 6m 448k 217k 128k 5k 21m 7m 11 1 4 1 674
The Vitals feature is only available as part of the SapMachine builds of OpenJDK. Thomas Stuefe, JVM engineer at SAP, indicated that the feature was not accepted upstream as its features overlap with Java Flight Recorder. More information can be found on GitHub and in the SapMachine Vitals blog.