How to: Managing System Logs with Journalctl
In some of the newer Linux distributions, “init.d” or “rsyslogd” has been replaced by a program called JournalCTL. Previously, logs were typically found in a particular location for each separate log file, rather than being easy to locate, manage, and manipulate. Systemd made these log files easier to access by centralizing them, logging all kernel and user processes in one “journal.” Journalctl is the newest version of this, which makes it even easier to manage these files.
In this guide, we will explain the basics of journalctl as well as a brief overview of suggested uses for the program.
Must have a Linux distribution with journalctl as the system log management program, for example, CentOS 7. If you do not have a server, consider a super fast SSD virtual private server from Atlantic.Net.
In This Article
Setting System Time
Before beginning with anything else, it is highly suggested to set the system time. Since systemd logs as a binary journal, you can log your records in either UTC or local (server) time. By default, systemd will display results in local time. To avoid confusion later on, we will check to ensure that the time is set correct. Systemd has a built in program called “timedatectl” that is for this purpose.
First, check to ensure your time zone is available in the zone options:
This shows all available timezones on your server. After locating the one matching your location (or your server location), set it:
sudo timedatectl set-timezone (zone)
Then we will check to ensure this took effect properly:
The first line will display the correct time.
By default, journalctl users without root privileges can only see their own log files. The system administrator can grant access to complete log files. Perform the following command while logged in as the root user
usermod -a -G adm username
Replace “username” with the name of the user. This user will receive the same journalctl access as the root user. However, this control only works when persistent storage is enabled.
Basic Log Viewing
There are a few commands that can be issued at this point. The basic command will display an output of all general entries, displaying oldest entries first:
Though this will give you all the log files, this will give you, well, all the log files. This means that you will have hundreds or thousands, or more, of log lines displayed if given the chance. This is similar to what you would see if you were to look in “/var/log/messages/” in previous versions.
Current Boot Session
We also have a command that will display logs that only occurred in the current boot.
This means that any information from previous boot sessions will not be shown.
Previous Boot Sessions
Should you need to review logs for previous boot sessions, there is an option for that as well. Some versions enable logging of previous boot sessions by default. To ensure that yours is set to log these, we suggest enabling this feature. This can be done by creating the directory
sudo mkdir -p /var/log/journal
or by editing the journal configuration file directly
sudo nano /etc/systemd/journald.conf
In the file, under [Journal], set the “Storage=” option to “persistent” so that persistent logging is enabled. Save and exit this file and the settings will take effect going forward.
Once this has been saved, to view available options for boot files to review,
You will see a line for each available boot session. The offset ID is the first column while the boot ID is the second column. You can use either the offset ID or boot ID to issue the command:
journalctl -b (offset/boot ID)
Specific Time Frame
You may, instead, wish to review the log files by a specified time frame, which may span several boot sessions or limit it to a specified time window within one boot session. The –since and –until limitations can be placed which will show only events after and prior to the specific log.
For all time values, the following format should be used:
If you would like to view the events that have happened since 10:00 AM on July 5th, 2015, we can achieve this by the following command:
journalctl --since “2015-07-05 10:00:00”
Please keep in mind that these will be based either on server time or UTC, depending on what you set in your preferences earlier in this tutorial.
If the date is omitted, the journal will assume today’s date. If the time is omitted, midnight (00:00:00) will be assumed. You also do not have to include the seconds field, in which case “00” seconds will be assumed.
There are also friendly commands, such as
journalctl --since yesterday
journalctl --since 02:00 --until “3 hours ago”
The most useful function available is the unit filtering. The -u filtering gives the option to view only a specific “unit” allowing you to see things that have happened only in reference to a specific program or systems. As always, you can add filters for time frames with this.
journalctl -u nginx.service
journalctl -u nginx.service --since yesterday
By Process, User, or Group ID
If you have already reviewed enough information to retrieve the PID of the process you are looking for, you can filter for this instead. This is run with _PID
Filtering based on path locations is also available. If you are selecting an executable path, journalctl will show all entries related to that executable. If, for example, you are looking for logs involving bash,
This method usually includes more detailed information on processes and child processes, when available. Sometimes this is not an option, however.
Kernel messages would typically be located in dmesg output, however we can use journalctl to retrieve these as well. This would be with the -k or -dmesg filter.
This will show kernel messages that have occurred during the current boot session. Again, this can always be modified with the previously mentioned options. If you wanted to review the kernel messages from the prior boot session,
journalctl -k -b -1
Priority is often the preferred method of reviewing logs. Journalctl can be used to display messages only above a certain priority level. The following is the priority levels:
You can use either the name of the priority level or the corresponding number. If you only wish to see logs of warning priority or above, you can issue the command
journalctl -p warning -b
journalctl -p 4 -b
To view the list of journals that occur within a specific field, use the -F modifier
journal -F fieldname
where “fieldname” is replaced by the field you are seeking. You also have the option of displaying only log entries that fit a specific condition
Truncate or Expand Output
Journalctl can be modified to shrink or expand output data display. By default, journalctl will display the entire entry, requiring you to scroll to the right with the arrow keys. We can truncate this output instead, which will show an ellipsis for removed information with the –no-full option.
Alternatively you can tell journalctl to display all of the information, regardless of it considering a character to be unprintable. This is done with the -a modifier.
Change to Standard Output
By default, journalctl shows output in a pager to make it easier to review. In the event of modifying or reviewing the information via text manipulation, it may be better and easier to use the standard output with the –no-pager modifier.
If you are using a text editor on journal entries, as we discussed above, the journal can be viewed in other formats as needed. This can be done with the -o modifier and a format specifier.
For example, you can read the journal in JSON by typing
journalctl -b -u nginx -o json
You can use the following for display options:
cat — Displays the message field itself
export — Binary format for transferring or backing up
json — standard JSON, one entry per line
json-pretty — JSON format but is easier to read
json-sso — JSON format wrapped to make add server-sent event compatible
short — Default syslog output
short-iso — Default output that shows ISO 8601 wallclock timestamps
short-monotonic — Default output with monotonic timestamp
short-precise — Default output with microsecond timestamp
verbose — Shows all journal fields for that entry, includes hidden
Active Process Monitoring
Journalctl imitates which administrators use tail for monitoring. This was built into journalctl allowing access to these features without having to pipe in additional tools.
Displaying Recent Logs
The -n option can be used to display a specific amount of records. Not including a modifying number will display ten (10) results, by default.
You can, instead, specify how many entries you would like to see
journalctl -n 20
The -f modifier can be used to follow logs as they are being written.
There are a few commands that can be used to clean up some older logs for the sake of your hard drive space.
First, you can find out just how much space is being taken up by journals by using the –disk-usage modifier.
There are two different ways to clean this up. The first is to shrink the journal based on the size. This removes entries from oldest to newest until the desired disk space is reached.
sudo journalctl --vacuum-size-2G
The alternate way to shrink the journal is to a certain date. The –vacuum-time option deletes any entries up to a certain date.
To keep entries only from last year, you can type
sudo journalctl --vacuum-time-1years
Limiting Journal Size
You have the ability to place limits on the amount of space journals can take up.
These changes can be made with any of the following field modifications
SystemMaxUse= Maximum disk space
SystemKeepFree= Amount of space that should be kept available
SystemMaxFileSize= How large individual journal files can become before being rotated
RuntimeMaxUse= Maximum disk space that can be used within volatile storage (/run filesystem)
RuntimeKeepFree= The amount of space that should be kept available for others in volatile storage (/run filesystem)
RuntimeMaxFilesize= How large individual journal file can become in volatile storage (within /run filesystem) before being rotated
Thank you for following this how-to. Please check back here for more updates, and to consider a market-leading VPS hosting server from Atlantic.Net.
Get a $250 Credit and Access to Our Free Tier!
Free Tier includes:
G3.2GB Cloud VPS a Free to Use for One Year
50 GB of Block Storage Free to Use for One Year
50 GB of Snapshots Free to Use for One Year