How to Use journalctl to Analyze Logs in Linux

Systemd is the default system manager in all major Linux-based operating systems. It provides a journald daemon that handles all messages produced by the kernel and system services. The journald daemon collects data from all available sources and stores them in a binary format for easy and dynamic manipulation. Systemd provides a command-line tool called journalctl that can be used to read and analyze journal logs. journalctl allows you to analyze and monitor the logs in real-time.

In this guide, we will show you how to use journalctl to analyze logs in Linux.

Configure Journal

By default, journal logs are enabled and stores log data at /run/log/journal/. But, since logs are deleted automatically after a system reboot, you will need to configure Journal to store all logs permanently.

First, create a directory to store the Journal log:

mkdir /var/log/journal

Next, set proper ownership with the following command:

chown -R root:systemd-journal /var/log/journal

Next, edit the journald default configuration file /etc/systemd/journald.conf and define your new directory:

nano /etc/systemd/journald.conf

Change the following line:

Storage=persistent

Save and close the file, then restart the systemd-journald service to apply the changes:

systemctl restart systemd-journald

You can now check the /var/log/journal directory:

ls -l /var/log/journal

You should see the following output:

drwxr-xr-x 2 root systemd-journal 4096 Apr 21 11:35 97bcb1f0d9aa4b339adefc87f1332d04

Use journalctl to Analyze Log

To print all logs collected by journald daemon, run the journalctl command:

journalctl

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:40:12 UTC. --
Apr 21 07:00:15 ubuntu2004 kernel: Linux version 4.19.0-9-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07)
Apr 21 07:00:15 ubuntu2004 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-9-amd64 root=UUID=d4f8c3a8-164f-4e15-a198-6124ce8719b5 ro
Apr 21 07:00:15 ubuntu2004 kernel: x86/fpu: x87 FPU will use FXSAVE
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-provided physical RAM map:
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x0000000000100000-0x000000007ffdbfff] usable
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x000000007ffdc000-0x000000007fffffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
Apr 21 07:00:15 ubuntu2004 kernel: NX (Execute Disable) protection: active

If you need more verbose output, run the following command:

journalctl -o verbose

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:40:29 UTC. --
Wed 2021-04-21 07:00:15.461318 UTC [s=1baac74dce14445f9a6670f231104955;i=1;b=41c491f449fa44c288474cf9f14386c0;m=1ee776;t=5c0761d6627c6;x=4c88a9
    _SOURCE_MONOTONIC_TIMESTAMP=0
    _TRANSPORT=kernel
    PRIORITY=5
    SYSLOG_FACILITY=0
    SYSLOG_IDENTIFIER=kernel
    MESSAGE=Linux version 4.19.0-9-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.118-2+deb10u1
    _BOOT_ID=41c491f449fa44c288474cf9f14386c0
    _MACHINE_ID=97bcb1f0d9aa4b339adefc87f1332d04
    _HOSTNAME=ubuntu2004
Wed 2021-04-21 07:00:15.461357 UTC [s=1baac74dce14445f9a6670f231104955;i=2;b=41c491f449fa44c288474cf9f14386c0;m=1ee79e;t=5c0761d6627ed;x=eaf7df
    _SOURCE_MONOTONIC_TIMESTAMP=0
    _TRANSPORT=kernel
    SYSLOG_FACILITY=0
    SYSLOG_IDENTIFIER=kernel
    _BOOT_ID=41c491f449fa44c288474cf9f14386c0
    _MACHINE_ID=97bcb1f0d9aa4b339adefc87f1332d04
    _HOSTNAME=ubuntu2004
    PRIORITY=6
    MESSAGE=Command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-9-amd64 root=UUID=d4f8c3a8-164f-4e15-a198-6124ce8719b5 ro

To list all available boot logs, run the following command:

journalctl --list-boots

Output:

 0 41c491f449fa44c288474cf9f14386c0 Wed 2021-04-21 07:00:15 UTC—Wed 2021-04-21 11:41:44 UTC

To display all logs since the most recent reboot, run the following command:

journalctl -b

To display the most recent log entries, run the following command:

journalctl --lines 5

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:45:13 UTC. --
Apr 21 11:45:06 ubuntu2004 sshd[12088]: Failed password for invalid user telecomadmin from 103.42.205.111 port 64471 ssh2
Apr 21 11:45:08 ubuntu2004 sshd[12088]: Connection closed by invalid user telecomadmin 103.42.205.111 port 64471 [preauth]
Apr 21 11:45:13 ubuntu2004 sshd[12092]: Invalid user admin from 81.70.161.162 port 60614
Apr 21 11:45:13 ubuntu2004 sshd[12092]: pam_unix(sshd:auth): check pass; user unknown
Apr 21 11:45:13 ubuntu2004 sshd[12092]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=81.70.161.162

To print the log continuously, run the following command:

journalctl --follow

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC. --
Apr 21 11:45:36 ubuntu2004 sshd[12106]: Disconnected from invalid user babi 104.131.102.169 port 54872 [preauth]
Apr 21 11:45:40 ubuntu2004 sshd[12108]: Invalid user telecomadmin from 103.108.241.111 port 60842
Apr 21 11:45:40 ubuntu2004 sshd[12108]: pam_unix(sshd:auth): check pass; user unknown
Apr 21 11:45:40 ubuntu2004 sshd[12108]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=103.108.241.111
Apr 21 11:45:41 ubuntu2004 sshd[12110]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=101.32.11.137  user=root
Apr 21 11:45:42 ubuntu2004 sshd[12108]: Failed password for invalid user telecomadmin from 103.108.241.111 port 60842 ssh2
Apr 21 11:45:43 ubuntu2004 sshd[12108]: Connection closed by invalid user telecomadmin 103.108.241.111 port 60842 [preauth]
Apr 21 11:45:43 ubuntu2004 sshd[12110]: Failed password for root from 101.32.11.137 port 43086 ssh2
Apr 21 11:45:45 ubuntu2004 sshd[12110]: Received disconnect from 101.32.11.137 port 43086:11: Bye Bye [preauth]
Apr 21 11:45:45 ubuntu2004 sshd[12110]: Disconnected from authenticating user root 101.32.11.137 port 43086 [preauth]

To display specific service-related logs like SSH and Nginx, run the following command:

journalctl -u ssh.service
journalctl -u nginx.service

To display only kernel-related logs, run the following command;

journalctl -k

To display logs which contain error or critical, run the following command:

journalctl -p err -b

Output:

-- Logs begin at Wed 2021-04-21 07:00:15 UTC, end at Wed 2021-04-21 11:49:21 UTC. --
Apr 21 07:00:16 ubuntu2004 ntpd[337]: leapsecond file ('/usr/share/zoneinfo/leap-seconds.list'): expired less than 115 days ago
Apr 21 07:00:16 ubuntu2004 ntpd[337]: bind(21) AF_INET6 fe80::200:2dff:fe3a:264e%2#123 flags 0x11 failed: Cannot assign requested address
Apr 21 07:00:16 ubuntu2004 ntpd[337]: unable to create socket on ens3 (5) for fe80::200:2dff:fe3a:264e%2#123

To display all logs from yesterday, run the following command:

journalctl --since yesterday

To display all logs starting from 6:00 AM and continuing until an hour ago, run the following command:

journalctl --since 06:00 --until "1 hour ago"

To display the amount of space used by the journal, run the following command:

journalctl --disk-usage

Output:

Archived and active journals take up 16.0M in the file system.

If you want to keep all logs’ data only from the last year, run the following command:

journalctl --vacuum-time=1years

To display only the last few logs, run the following command:

journalctl -xe

Conclusion

In the above guide, you learned how to use journalctl to read and analyze different system logs. You can now identify or troubleshoot system or application-related issues on your VPS from Atlantic.Net.