Skip to content

Journald - logging system

Introduction

Journald is a central logging and log management system, being an integral part of the systemd ecosystem - currently the dominant init system in most Linux distributions.

Integration with systemd

One of the advantages of journald is its integration with systemd. As a native component of this ecosystem, journald:

  • Starts almost simultaneously with the init system, meaning that practically all messages generated during system startup (even very early ones) are captured and recorded
  • Has access to systemd metadata about units that generate logs
  • Automatically tags messages with additional information, such as service ID, user, or group
  • Allows filtering logs by systemd-specific metadata

Thanks to this connection, journald offers a more coherent and complete picture of system operation than traditional solutions.

Binary format - performance and structure

Unlike traditional logging systems that store logs as plain text files, journald uses a binary format for data storage. This approach brings several benefits:

Advantages of binary format

  • Higher performance - I/O operations are much faster than with text files
  • Lower space usage - binary format allows for efficient data compression
  • Indexing - built-in indexes speed up log searching and filtering
  • Structuring - each entry contains a set of fields with metadata, facilitating later analysis

Structure of a journald entry

Each entry in journald consists of:

  • Timestamp with high precision (to microsecond)
  • Priority of the message (according to traditional syslog classification)
  • Source identifier (e.g., service name, process)
  • UID/GID identifier of user and group
  • PID identifier of the process
  • Actual message
  • Many additional metadata (hostname, systemd unit, etc.)

All this information is available for each entry, allowing for very precise filtering and analysis.

Easy parsing and data export

Despite the binary format, journald offers convenient tools for working with logs:

Parsing tools

The basic tool is the journalctl command, which offers extensive options:

Display all logs

journalctl

From the last 10 minutes

journalctl --since "10 minutes ago"

Only for a specific service

journalctl -u nginx.service

From a specific time range

journalctl --since "2025-01-01" --until "2025-01-02"

Logs related to a specific process

journalctl _PID=1234

JSON export

Journald enables easy data export to JSON format, which is particularly useful for automation and integration with other tools:

Export all logs to JSON format

journalctl -o json

Export with additional filtering options

journalctl -u nginx.service -o json-pretty

Export to file

journalctl -o json > logs.json

JSON format preserves all metadata, making it ideal for further processing by scripts or log analysis systems.

Relationship with traditional syslog (rsyslog)

Journald and traditional rsyslog are not mutually exclusive systems - in most modern distributions, they work together.

Journald cooperation with rsyslog

Typical configuration in modern distributions looks as follows:

  1. Journald acts as the first log receiver, capturing all system messages
  2. Rsyslog can be configured to read logs from journald and additionally write them in traditional text format

Such architecture ensures:

  • Complete log capture by journald, with all metadata
  • Backward compatibility through writing in traditional text files
  • Ability to use both modern and traditional log analysis tools

Differences and advantages of journald over rsyslog

Feature Journald Rsyslog
Storage format Binary, structured Text
Metadata Rich, numerous fields Limited
Indexing Built-in None
Log rotation Automatic Requires separate configuration
Compression Automatic Requires separate configuration
Integrity protection Built-in Requires additional tools
Starts alongside init Yes No

Integration with log analysis tools

Journald integrates well with popular log aggregation and analysis systems.

Splunk

Splunk is one of the most popular commercial log analysis tools. Integration with journald can be implemented in several ways:

  1. Through Universal Forwarder - Splunk UF can be configured to read data from journald using an adapter
  2. Using scripts - exporting logs from journald to JSON and sending to Splunk via HTTP Event Collector
  3. Using rsyslog as intermediary - rsyslog reads logs from journald and forwards them to Splunk

Example Splunk Universal Forwarder configuration for reading journald:

[monitor:///var/log/journal]
index = linux_logs
sourcetype = journald

Logstash (ELK Stack)

Logstash, as part of the Elasticsearch-Logstash-Kibana (ELK) stack, offers ready solutions for working with journald logs:

  1. Journald plugin - dedicated plugin for direct reading of data from journald
  2. Filebeat - can be configured to monitor exported data

Example Logstash configuration with journald plugin:

input {
    journald {
        path => "/var/log/journal"
        seekto => "tail"
        thisboot => true
    }
}

filter {
    if [_systemd_unit] == "nginx.service" {
        mutate { add_tag => "nginx" }
    }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "journald-%{+YYYY.MM.dd}"
    }
}

Graylog

Graylog can retrieve data from journald through:

  1. Sidecar with configurable collectors
  2. Forwarding via syslog with appropriate rsyslog configuration

Example commands

Debugging system boot problems

Display logs from last system boot

journalctl -b

Display logs from specific boot moment

journalctl -b -1

Logs with time stamps (for boot performance analysis)

journalctl -b --output=short-monotonic

Monitoring specific services

Continuous service monitoring (like tail -f)

journalctl -f -u nginx.service

Logs of error level messages and higher

journalctl -p err -u mysql.service

Security event analysis

Authorization logs

journalctl _COMM=sshd

System audit logs

journalctl _AUDIT_TYPE=1107

Challenges and limitations

Despite many advantages, journald also has some limitations:

  • Journal size - by default limited to 10% of free disk space
  • Compatibility - older tools may not work directly with journald
  • Complexity - more factors to consider when configuring logging infrastructure

Summary

Journald represents a modern approach to logging in Linux systems, offering:

  • Integration with systemd
  • Efficient binary storage format
  • Rich metadata and advanced filtering capabilities
  • Easy integration with log analysis systems

For most deployments, combining journald (for efficient collection and storage) with traditional rsyslog (for compatibility) and log analysis tools (like Splunk or ELK) provides an optimal solution, ensuring both efficiency and flexibility.

In the next section, you'll find detailed information about configuring the logging service itself in journald and all (known to me) available options.

Enjoy!