Simple Self-hosted Log Navigation

There are a bunch of ways to deal with the plethora of logs when self-hosting or running a homelab.

And, some are more sensible than others.

As with everything Linux setup, there are simple ways, complicated ways, overly complicated ways, the enterprise ways and the complicated enterprise ways.

If you’ve got a low power requirement or low headache requirement, you may prefer the simplest, most native approach when dealing with non-critical systems like a homelab and self-hosted services.

i.e. I’d rather not host an application for logging when there are some great native alternatives.

Here’s some groundwork information and a few great tools that will get you get started with logging. I’m going to cover:

• logging basics on Linux
• lnav

Linux Logging Basics

We’re talking about Linux systems here and although I’ll wish you the best of luck if you’re self-hosting on anything else, this article probably only applies to GNU/Linux like systems.

There are a bunch of logfiles and log formats built into most distros like Debian, Fedora and Ubuntu etc.

The most common logfiles you’ll come across are stored under /var/log/:

• syslog
• apache/nginx access log
• apache/nginx/php error log
• JSON array dumps

Logs on Linux are typically “rotated” to prevent the files becoming too large and unwieldy. This is often handled with the “logrotate” programme.

• auth.log would be the most recent logfile
• auth.log.1 would be the next most recent logfile
• etc

Incrementing the number until archiving kicks in, which is specified in the logrotate configuration. This will typically compress the file, resulting in a filename similar to this; auth.log.4.gz.

Due to the repetitive and text-based nature of logfiles, they lend themselves well to effective compression. You can end up keeping a lot of them before running into trouble.

After a specified period of time, the old and likely compressed log files are usually deleted to prevent them from accumulating and eventually using every last bit of space on the system.

Syslog Logfiles

…and by extension, systemd’s journald.

To view syslog files, you’ll likely need to be logged in as root or using sudo. System level logs regularly contain somewhat sensitive information and so, even the read access permissions are limited.

Syslogs are likely to be your first stop for everything happening on your system. Check out the follow default log files; kern, auth, cron etc to get started with the basics.

Access and Error Web-server Logs

Most of the common web/https servers that serve content from /var/www or /srv output hit logs in one of a few standard “access log” formats.

In these logs you’ll find the information about who’s connecting to your server, how they’re connected and what they’re up to.

You’ll also likely come across “error_log” files for server side errors and warnings from languages like PHP that are running through the httpd server of choice.

Docker Logs

Docker (the non-desktop application installed from either docker-ce or docker.io packages on Debian, the latter being my preferred) tends to do things its own away rather than the expected way you may be used to in a GNU/Linux system. This is not too dissimilar to how systemd handles itself.

At the time of writing 2023-11-27, docker uses its own json log format. It essentially dumps the entire output of data from a container (usually the stdout and stderr) straight into a single JSON file.

This is not a wonderful method, it’s likely to become unstable without rotation, and since JSON is not designed for logging, it can be quite cumbersome to review without some relatively heavy post-processing.

Docker has started to adopt the local log type, but this is not as clean as it might seem. It’s actually an internal system to docker, and interacting with the files natively is unfortunately discouraged.

However, all is not lost! You can push docker container logs through to syslog or even a remote server such as rsyslog. Then you’ll be able to handle the logs much more sensibly.

I’ll hopefully write about setting up rsyslog with SSH tunnels as a secure centralised syslog server in the future.

To keep docker logs manageable I’d recommend spending some time to make sure your containers are individually configured, to generate only the level of logging you need. There is often a lot of output information that’s not usually needed outside of debugging.

Viewing Logs on Linux

You can always view a log file on Linux directly.

e.g. use cat to output the entire file, better still less to output a chunk of the file (i.e. a pager) or even better still tail -f to follow the live end of the logfile.

e.g. sudo tail -f /var/log/auth.log will allow you to follow along with the authentication logs generated in the syslog format.

However, these methods of reviewing large logs files quickly get bland and overwhelming. (…perhaps in a similar way to a blog post about logging?)

To make it a bit easier to spot the problematic log entries we can get a little fancier with lnav. It’s not quite as fancy as full logging services like Graylog, Grafana Loki etc, but it’ll work for most log reviewing tasks.

“Log Navigator” or lnav is a great TUI application. It’s a super simple, and native tool for browsing all sorts of log formats (expect Dockers rather dumb JSON format) in colour codes style. It’s got some powerful features for navigation back and forth through important log entries while skipping the unimportant.

Although their site and Github would suggest you install the binary directly from their repo for the most up-to-date version, I’m quite happy installing a trusted and older version from the Debian repository.

sudo apt install lnav

Have a play around with it on some basic syslog files on your Linux system. Use the “?” key to open the help dialogue.

I’ll also throw in an honourable mention for goaccess. It’s an Apache access log style browser modelled around analytics over time, somewhat similar to Google Analytics but only using the access logs.

Let me know if you’ve got any tips yourself!

Comments & Questions

Reply by email to send in your thoughts.

Comments may be featured here unless you say otherwise. You can encrypt emails with PGP too, learn more about my email replies here.

PGP: 9ba2c5570aec2933970053e7967775cb1020ef23