Understanding journalctl in Linux

Understanding journalctl in Linux

journalctl is a command-line utility used to query and display logs managed by systemd’s journal. It provides powerful methods for viewing, filtering, and analyzing logs in Linux systems that use systemd for managing services and processes. The journal logs are stored in a binary format, which allows efficient filtering by time, service, or severity.

Here's a complete guide on how to use journalctl, its features, and useful methods for analyzing logs.

1. Basic Usage of journalctl

By default, running journalctl without any options will display the entire system journal since boot, starting from the oldest log entries. The logs shown include system messages, service logs, kernel messages, and user session logs.

Example:

journalctl

2. Viewing Logs in Real-Time (Follow Mode)

If you want to watch the logs in real-time as new entries are added, similar to the tail -f command, you can use the -f option.

Example:

journalctl -f

This is useful for monitoring logs when you are debugging or waiting for specific events to happen.

3. Filtering Logs by Time

One of journalctl's powerful features is the ability to filter logs by a specific time or range.

Viewing logs from today:

journalctl --since today

Viewing logs from a specific date/time:

journalctl --since "2024-09-18 14:00:00" --until "2024-09-18 15:00:00"

Viewing logs starting from a relative time:

journalctl --since "1 hour ago"

4. Filtering by Service or Unit

In systemd, services and processes are grouped into units, such as systemd.service for system services or systemd.timer for timers. You can filter logs related to a specific service by using the -u option.

Example: View logs for a specific service (e.g., nginx.service):

journalctl -u nginx.service

This will show logs only related to the nginx service. You can combine this with other options like time filters:

journalctl -u nginx.service --since yesterday

5. Viewing Kernel Logs

To display only kernel-related logs, you can use the -k option. Kernel logs typically contain information about hardware, boot processes, and kernel-level events.

Example:

journalctl -k

You can also view real-time kernel logs:

journalctl -kf

6. Filtering by Priority (Log Levels)

Logs in systemd’s journal are categorized by priority (or severity). These priorities are represented by numeric values:

• 0: Emergency
• 1: Alert
• 2: Critical
• 3: Error
• 4: Warning
• 5: Notice
• 6: Informational
• 7: Debug

You can filter logs by their priority level using the -p option.

Example: Show only error-level or more severe logs:

journalctl -p err

This command will display logs with priority 3 (Error) and above.

7. Limiting the Number of Log Entries

If you want to limit the number of entries displayed, use the -n option followed by the number of lines you want to view.

Example: Show the last 20 log entries:

journalctl -n 20

You can combine this with other options, such as following the logs in real-time:

journalctl -n 20 -f

8. Filtering by User or Process ID (PID)

You can filter logs based on the user ID (UID) or process ID (PID) to track down logs specific to a user or a process.

Example: Show logs for a specific user (e.g., UID 1000):

journalctl _UID=1000

Example: Show logs for a specific process (e.g., PID 1234):

journalctl _PID=1234

9. Filtering by Specific Fields

journalctl supports filtering logs by various metadata fields, which can include process name, executable path, systemd unit, and more.

Some common fields include:

• _COMM: Command name.
• _EXE: Executable path.
• _SYSTEMD_UNIT: Systemd unit name.
• _UID: User ID.

Example: Filter logs by executable name:

journalctl _COMM=my_program

10. Exporting and Saving Logs

You can export logs to a file in different formats for later analysis or debugging. Use the --output option to specify the output format, and redirect the output to a file.

Example: Export logs in plain text format:

journalctl --since today --output short > logs.txt

Example: Export logs in JSON format:

journalctl --since today --output json > logs.json

This is useful if you need to parse or analyze logs programmatically.

11. Clearing the Journal Logs

If journal logs are consuming too much disk space, you can free up space by clearing old logs or limiting the size of the journal.

Clear all journal logs:

sudo journalctl --vacuum-time=2weeks

This will remove logs older than 2 weeks.

Limit journal log size to 1GB:

sudo journalctl --vacuum-size=1G

12. Persistent Journal Configuration

By default, journalctl logs may only be stored in memory (volatile storage), which means they are lost after a reboot. To make the logs persistent across reboots, you can enable persistent logging by configuring the journal storage settings.

Steps:

Restart the systemd-journald service:

sudo systemctl restart systemd-journald

Modify the systemd journal configuration:
Edit /etc/systemd/journald.conf and set:

[Journal]
Storage=persistent

Create the journal directory:

sudo mkdir -p /var/log/journal

This will store logs persistently in /var/log/journal.

journalctl is a powerful utility for managing and analyzing systemd logs on Linux systems. It offers a wide range of features, including filtering by time, service, priority, and more. By using the methods described here, you can efficiently monitor, troubleshoot, and manage logs in a systemd-based Linux environment.