How to Use journalctl to Analyze Logs in Linux

Systemd is the default system manager in all major Linux-based operating systems. It provides a journald daemon that handles all messages produced by the kernel and system services. The journald daemon collects data from all available sources and stores them in a binary format for easy and dynamic manipulation. Systemd provides a command-line tool called journalctl that can be used to read and analyze journal logs. journalctl allows you to analyze and monitor the logs in real-time.

In this guide, we will show you how to use journalctl to analyze logs in Linux.

Prerequisites

• A fresh Ubuntu 20.04 server on the Atlantic.Net Cloud Platform
• A root password configured on your server

Create Atlantic.Net Cloud Server

First, log in to your Atlantic.Net Cloud Server. Create a new server, choosing Ubuntu 20.04 as the operating system with at least 2GB RAM. Connect to your Cloud Server via SSH and log in using the credentials highlighted at the top of the page.

Once you are logged in to your Ubuntu 20.04 server, run the following command to update your base system with the latest available packages.

apt-get update -y

Configure Journal

By default, journal logs are enabled and stores log data at /run/log/journal/. But, since logs are deleted automatically after a system reboot, you will need to configure Journal to store all logs permanently.

First, create a directory to store the Journal log:

mkdir /var/log/journal

Next, set proper ownership with the following command:

chown -R root:systemd-journal /var/log/journal

Next, edit the journald default configuration file /etc/systemd/journald.conf and define your new directory:

nano /etc/systemd/journald.conf

Change the following line:

Storage=persistent

Save and close the file, then restart the systemd-journald service to apply the changes:

systemctl restart systemd-journald

You can now check the /var/log/journal directory:

ls -l /var/log/journal

You should see the following output:

drwxr-xr-x 2 root systemd-journal 4096 Apr 21 11:35 97bcb1f0d9aa4b339adefc87f1332d04

Use journalctl to Analyze Log

To print all logs collected by journald daemon, run the journalctl command:

journalctl

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:40:12 UTC. --
Apr 21 07:00:15 ubuntu2004 kernel: Linux version 4.19.0-9-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07)
Apr 21 07:00:15 ubuntu2004 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-9-amd64 root=UUID=d4f8c3a8-164f-4e15-a198-6124ce8719b5 ro
Apr 21 07:00:15 ubuntu2004 kernel: x86/fpu: x87 FPU will use FXSAVE
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-provided physical RAM map:
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x0000000000100000-0x000000007ffdbfff] usable
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x000000007ffdc000-0x000000007fffffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: NX (Execute Disable) protection: active

If you need more verbose output, run the following command:

journalctl -o verbose

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:40:29 UTC. --
Wed 2021-04-21 07:00:15.461318 UTC [s=1baac74dce14445f9a6670f231104955;i=1;b=41c491f449fa44c288474cf9f14386c0;m=1ee776;t=5c0761d6627c6;x=4c88a9
    _SOURCE_MONOTONIC_TIMESTAMP=0
    _TRANSPORT=kernel
    PRIORITY=5
    SYSLOG_FACILITY=0
    SYSLOG_IDENTIFIER=kernel
    MESSAGE=Linux version 4.19.0-9-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.118-2+deb10u1
    _BOOT_ID=41c491f449fa44c288474cf9f14386c0
    _MACHINE_ID=97bcb1f0d9aa4b339adefc87f1332d04
    _HOSTNAME=ubuntu2004
Wed 2021-04-21 07:00:15.461357 UTC [s=1baac74dce14445f9a6670f231104955;i=2;b=41c491f449fa44c288474cf9f14386c0;m=1ee79e;t=5c0761d6627ed;x=eaf7df
    _SOURCE_MONOTONIC_TIMESTAMP=0
    _TRANSPORT=kernel
    SYSLOG_FACILITY=0
    SYSLOG_IDENTIFIER=kernel
    _BOOT_ID=41c491f449fa44c288474cf9f14386c0
    _MACHINE_ID=97bcb1f0d9aa4b339adefc87f1332d04
    _HOSTNAME=ubuntu2004
    PRIORITY=6
    MESSAGE=Command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-9-amd64 root=UUID=d4f8c3a8-164f-4e15-a198-6124ce8719b5 ro

To list all available boot logs, run the following command:

journalctl --list-boots

Output:

 0 41c491f449fa44c288474cf9f14386c0 Wed 2021-04-21 07:00:15 UTC—Wed 2021-04-21 11:41:44 UTC

To display all logs since the most recent reboot, run the following command:

journalctl -b

To display the most recent log entries, run the following command:

journalctl --lines 5

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:45:13 UTC. --
Apr 21 11:45:06 ubuntu2004 sshd[12088]: Failed password for invalid user telecomadmin from 103.42.205.111 port 64471 ssh2
Apr 21 11:45:08 ubuntu2004 sshd[12088]: Connection closed by invalid user telecomadmin 103.42.205.111 port 64471 [preauth]
Apr 21 11:45:13 ubuntu2004 sshd[12092]: Invalid user admin from 81.70.161.162 port 60614
Apr 21 11:45:13 ubuntu2004 sshd[12092]: pam_unix(sshd:auth): check pass; user unknown
Apr 21 11:45:13 ubuntu2004 sshd[12092]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=81.70.161.162

To print the log continuously, run the following command:

journalctl --follow

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC. --
Apr 21 11:45:36 ubuntu2004 sshd[12106]: Disconnected from invalid user babi 104.131.102.169 port 54872 [preauth]
Apr 21 11:45:40 ubuntu2004 sshd[12108]: Invalid user telecomadmin from 103.108.241.111 port 60842
Apr 21 11:45:40 ubuntu2004 sshd[12108]: pam_unix(sshd:auth): check pass; user unknown
Apr 21 11:45:40 ubuntu2004 sshd[12108]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.108.241.111
Apr 21 11:45:41 ubuntu2004 sshd[12110]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=101.32.11.137  user=root
Apr 21 11:45:42 ubuntu2004 sshd[12108]: Failed password for invalid user telecomadmin from 103.108.241.111 port 60842 ssh2
Apr 21 11:45:43 ubuntu2004 sshd[12108]: Connection closed by invalid user telecomadmin 103.108.241.111 port 60842 [preauth]
Apr 21 11:45:43 ubuntu2004 sshd[12110]: Failed password for root from 101.32.11.137 port 43086 ssh2
Apr 21 11:45:45 ubuntu2004 sshd[12110]: Received disconnect from 101.32.11.137 port 43086:11: Bye Bye [preauth]
Apr 21 11:45:45 ubuntu2004 sshd[12110]: Disconnected from authenticating user root 101.32.11.137 port 43086 [preauth]

To display specific service-related logs like SSH and Nginx, run the following command:

journalctl -u ssh.service
journalctl -u nginx.service

To display only kernel-related logs, run the following command;

journalctl -k

To display logs which contain error or critical, run the following command:

journalctl -p err -b

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:49:21 UTC. --
Apr 21 07:00:16 ubuntu2004 ntpd[337]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): expired less than 115 days ago
Apr 21 07:00:16 ubuntu2004 ntpd[337]: bind(21) AF_INET6 fe80::200:2dff:fe3a:264e%2#123 flags 0x11 failed: Cannot assign requested address
Apr 21 07:00:16 ubuntu2004 ntpd[337]: unable to create socket on ens3 (5) for fe80::200:2dff:fe3a:264e%2#123

To display all logs from yesterday, run the following command:

journalctl --since yesterday

To display all logs starting from 6:00 AM and continuing until an hour ago, run the following command:

journalctl --since 06:00 --until "1 hour ago"

To display the amount of space used by the journal, run the following command:

journalctl --disk-usage

Output:

Archived and active journals take up 16.0M in the file system.

If you want to keep all logs’ data only from the last year, run the following command:

journalctl --vacuum-time=1years

To display only the last few logs, run the following command:

journalctl -xe

Conclusion

In the above guide, you learned how to use journalctl to read and analyze different system logs. You can now identify or troubleshoot system or application-related issues on your VPS from Atlantic.Net.