Open-source log viewer hl is designed for efficient processing of structured logs in JSON or logfmt format. Built in Rust, it provides fast indexing and parsing, enabling to scan very large log files quickly, whether they are uncompressed or compressed.
According to benchmarks published by hl’s creator, the viewer achieves throughput of up to ~2 GiB/s with automatic indexing on initial scan and up to ~10 GiB/s when reindexing growing files. This performance appears to be a significant improvement over alternatives such as hlogf, humanlog, fblog, and fblog-d, making hl a compelling tool for DevOps engineers who work with very large log files from the command line.
The author says that hl also excels at processing "hundreds of local files totaling hundreds of gigabytes".
hl includes an integrated pager, like less, that makes it easy to browse log entries, along with a rich set of features for filtering, searching, sorting, and formatting logs. You can filter by log level (e.g., error, warn, info, and more), by field or key-value pair (e.g., component=tsdb, or request.method?!=GET, etc.), as well as using time ranges (e.g., --since yesterday, --since -3h, --until 'Jun 19 11:22:33', etc.)
Other useful features supported by hl include a follow mode, similar to tail -f, with automatic timestamp-based sorting across multiple sources using the -F option; customizable output with simple control for hiding or revealing specific fields using the -h option; automatic timezone switching to have timestamps converted to the desired timezone; and support for UI themes and color schemes.
A powerful capability in hl is its ability to build complex queries using logical, comparison, set-based, and string operators to combine basic filtering options. For example, the following command combines a log level and two field filters:
hl my-service.log --query 'level > info or status-code >= 400 or duration > 0.5'
Queries in hl also allow you to explicitly express how to deal with missing fields. For example, hl my-service.log -q 'exists(.price)' matches log entries where the price is defined, regardless of its value, excluding those where it is missing. Alternatively, hl my-service.log -q '.price?=3' returns entries where the price is either missing or equals 3.
In a recent Hacker News thread on the complexities of Grafana configuration, user solatic observed that for simple scenarios with only one instance of an app, hl is a good alternative to more complex tools like Grafan Loki, while also adding that:
If you have multiple machines, sending your logs to one place may not be necessary, but it's incredibly helpful; the alternative is basically ssh multiplexing.
hl is available under the MIT license and runs on macOS, Linux, and Windows.