How to: Managing System Logs with Journalctl

Introduction

In some of the newer Linux distributions, “init.d” or “rsyslogd” has been replaced by a program called JournalCTL.  Previously, logs were typically found in a particular location for each separate log file, rather than being easy to locate, manage, and manipulate.  Systemd made these log files easier to access by centralizing them, logging all kernel and user processes in one “journal.”  Journalctl is the newest version of this, which makes it even easier to manage these files.

In this guide, we will explain the basics of journalctl as well as a brief overview of suggested uses for the program.

Prerequisites

Must have a Linux distribution with journalctl as the system log management program, for example, CentOS 7.  If you do not have a server, consider a super fast SSD virtual private server from Atlantic.Net.

Setting System Time

Before beginning with anything else, it is highly suggested to set the system time.  Since systemd logs as a binary journal, you can log your records in either UTC or local (server) time.  By default, systemd will display results in local time.  To avoid confusion later on, we will check to ensure that the time is set correct.  Systemd has a built in program called “timedatectl” that is for this purpose.

First, check to ensure your time zone is available in the zone options:

timedatectl list-timezones

This shows all available timezones on your server.  After locating the one matching your location (or your server location), set it:

sudo timedatectl set-timezone (zone)

Then we will check to ensure this took effect properly:

timedatectl status

The first line will display the correct time.

Access Control

By default, journalctl users without root privileges can only see their own log files.  The system administrator can grant access to complete log files.  Perform the following command while logged in as the root user

usermod -a -G adm username

Replace “username” with the name of the user.  This user will receive the same journalctl access as the root user.  However, this control only works when persistent storage is enabled.

Basic Log Viewing

There are a few commands that can be issued at this point. The basic command will display an output of all general entries, displaying oldest entries first:

journalctl

Though this will give you all the log files, this will give you, well, all the log files.  This means that you will have hundreds or thousands, or more, of log lines displayed if given the chance.  This is similar to what you would see if you were to look in “/var/log/messages/” in previous versions.

Current Boot Session

We also have a command that will display logs that only occurred in the current boot.

journalctl -b

This means that any information from previous boot sessions will not be shown.

Previous Boot Sessions

Should you need to review logs for previous boot sessions, there is an option for that as well.  Some versions enable logging of previous boot sessions by default.  To ensure that yours is set to log these, we suggest enabling this feature.  This can be done by creating the directory

sudo mkdir -p /var/log/journal

or by editing the journal configuration file directly

sudo nano /etc/systemd/journald.conf

In the file, under [Journal], set the “Storage=” option to “persistent” so that persistent logging is enabled.  Save and exit this file and the settings will take effect going forward.
Once this has been saved, to view available options for boot files to review,

journalctl --list-boots

You will see a line for each available boot session.  The offset ID is the first column while the boot ID is the second column.  You can use either the offset ID or boot ID to issue the command:

journalctl -b (offset/boot ID)

Specific Time Frame

You may, instead, wish to review the log files by a specified time frame, which may span several boot sessions or limit it to a specified time window within one boot session.  The –since and –until limitations can be placed which will show only events after and prior to the specific log.
For all time values, the following format should be used:

YYYY-MM-DD HH:MM:SS

If you would like to view the events that have happened since 10:00 AM on July 5th, 2015, we can achieve this by the following command:

journalctl --since “2015-07-05 10:00:00”

Please keep in mind that these will be based either on server time or UTC, depending on what you set in your preferences earlier in this tutorial.
If the date is omitted, the journal will assume today’s date.  If the time is omitted, midnight (00:00:00) will be assumed.  You also do not have to include the seconds field, in which case “00” seconds will be assumed.

There are also friendly commands, such as

journalctl --since yesterday

and

journalctl --since 02:00 --until “3 hours ago”

Filtering Options

By Unit

The most useful function available is the unit filtering.  The -u filtering gives the option to view only a specific “unit” allowing you to see things that have happened only in reference to a specific program or systems.  As always, you can add filters for time frames with this.

journalctl -u nginx.service

or

journalctl -u nginx.service --since yesterday

By Process, User, or Group ID

If you have already reviewed enough information to retrieve the PID of the process you are looking for, you can filter for this instead.  This is run with _PID

journalctl _PID=4345

By Location

Filtering based on path locations is also available.  If you are selecting an executable path, journalctl will show all entries related to that executable.  If, for example, you are looking for logs involving bash,

journalctl /usr/bin/bash

This method usually includes more detailed information on processes and child processes, when available.  Sometimes this is not an option, however.

Kernel Messages

Kernel messages would typically be located in dmesg output, however we can use journalctl to retrieve these as well.  This would be with the -k or -dmesg filter.

journalctl -k

This will show kernel messages that have occurred during the current boot session.  Again, this can always be modified with the previously mentioned options.  If you wanted to review the kernel messages from the prior boot session,

journalctl -k -b -1

Priority

Priority is often the preferred method of reviewing logs.  Journalctl can be used to display messages only above a certain priority level.  The following is the priority levels:

0: emerg

1: alert

2: crit

3: err

4: warning

5: notice

6: info

7: debug

You can use either the name of the priority level or the corresponding number.  If you only wish to see logs of warning priority or above, you can issue the command

journalctl -p warning -b

or

journalctl -p 4 -b

Specified Field

To view the list of journals that occur within a specific field, use the -F modifier

journal -F fieldname

where “fieldname” is replaced by the field you are seeking.  You also have the option of displaying only log entries that fit a specific condition

journalctl fieldname=value

Truncate or Expand Output

Journalctl can be modified to shrink or expand output data display.  By default, journalctl will display the entire entry, requiring you to scroll to the right with the arrow keys.  We can truncate this output instead, which will show an ellipsis for removed information with the –no-full option.

journalctl --no-full

Alternatively you can tell journalctl to display all of the information, regardless of it considering a character to be unprintable.  This is done with the -a modifier.

journalctl -a

Change to Standard Output

By default, journalctl shows output in a pager to make it easier to review.  In the event of modifying or reviewing the information via text manipulation, it may be better and easier to use the standard output with the –no-pager modifier.

journalctl --no-pager

Output Formats

If you are using a text editor on journal entries, as we discussed above, the journal can be viewed in other formats as needed.  This can be done with the -o modifier and a format specifier.

For example, you can read the journal in JSON by typing

journalctl -b -u nginx -o json

You can use the following for display options:

cat — Displays the message field itself

export — Binary format for transferring or backing up

json — standard JSON, one entry per line

json-pretty — JSON format but is easier to read

json-sso — JSON format wrapped to make add server-sent event compatible

short — Default syslog output

short-iso — Default output that shows ISO 8601 wallclock timestamps

short-monotonic — Default output with monotonic timestamp

short-precise — Default output with microsecond timestamp

verbose — Shows all journal fields for that entry, includes hidden

Active Process Monitoring

Journalctl imitates which administrators use tail for monitoring.  This was built into journalctl allowing access to these features without having to pipe in additional tools.

Displaying Recent Logs

The -n option can be used to display a specific amount of records.  Not including a modifying number will display ten (10) results, by default.

journalctl -n

You can, instead, specify how many entries you would like to see

journalctl -n 20

Following Logs

The -f modifier can be used to follow logs as they are being written.

journalctl -f

Journal Maintenance

There are a few commands that can be used to clean up some older logs for the sake of your hard drive space.

First, you can find out just how much space is being taken up by journals by using the –disk-usage modifier.

journalctl --disk-usage

There are two different ways to clean this up.  The first is to shrink the journal based on the size.  This removes entries from oldest to newest until the desired disk space is reached.

sudo journalctl --vacuum-size-2G

The alternate way to shrink the journal is to a certain date.  The –vacuum-time option deletes any entries up to a certain date.

To keep entries only from last year, you can type

sudo journalctl --vacuum-time-1years

Limiting Journal Size

You have the ability to place limits on the amount of space journals can take up.

nano /etc/systemd/journald.conf

These changes can be made with any of the following field modifications

SystemMaxUse= Maximum disk space

SystemKeepFree= Amount of space that should be kept available

SystemMaxFileSize= How large individual journal files can become before being rotated

RuntimeMaxUse= Maximum disk space that can be used within volatile storage  (/run filesystem)

RuntimeKeepFree= The amount of space that should be kept available for others in volatile storage (/run filesystem)

RuntimeMaxFilesize= How large individual journal file can become in volatile storage (within /run filesystem) before being rotated

 

Thank you for following this how-to. Please check back here for more updates, and to consider a market-leading VPS hosting server from Atlantic.Net.