How to: Using lsof – 10 Essential Commands To Troubleshoot Your Linux Server

Verified and Tested 10/30/15

Introduction

At some point in their career, everyone is going to troubleshoot an issue. And everyone has their favorite troubleshooting step to use. For a lot of system admins, the first step to do is make sure that everything is up-to-date. For others it’s only a matter of checking log files before finding your solution. For me and others here at Atlantic.Net, it’s lsof; a command line utility used to list information about files that are opened by various processes.

Mostly used for troubleshooting network connection issues, lsof is a powerful yet too-little-known application. If you’re familiar with general Linux commands, lsof is an easy tool to remember because it “lists open files”.

lsof Usage Flags

One thing that’s important to understand is that lsof has a large number of available flags that can be used to modify the output you receive. Below shows all of the available flags:

[root@OffsiteLinux ~]# lsof --help
    usage: [-?abhlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-f[gG]] [+|-e s]
    [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s]
    [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]

Going through each flag would be a much longer article, so we’ll keep things simple and cover only the most used day-to-day options.

Key Option Flags

It’s important to understand how lsof works, especially by default. When you are passing options to the lsof command the default behavior is to OR the results. This means that running lsof using both the -i as well as the -p flag you will get the results of both.

These are the most often used flags in our troubleshooting:

default : with no options flags, lsof lists all open files for active processes
grouping : it's possible to group options, such as passing the flags though -nPi. Just keep an eye out for which options take additional parameters
-a : AND the result (instead of the default OR)
-l : show the userID instead of the username in the output
-h : get help
-i : selects the listing of files that match
-n : stop host name resolution of network numbers
-t : get process IDs only
-P : stop conversion of port numbers to port names

Getting Started

List all open files.

# lsof

List all open files.

Show all connections:

# lsof -i

Show all connections

The server we’re testing on is a new Atlantic.Net CentOS6 server which does not have any additional software packages installed. You can see from the above output that the server is only listening for SSH and SMTP traffic. If you want to see the actual port number instead of the service name we will need to run the command using the -Pi flag:

lsof -Pi

Show all connections listening on ports without resolving service name.

As you can see, all of the *:ssh and localhost:smtp output has been converted to *:22 and localhost:25. For most system administrators, especially if you’re not used to going through and analyzing system services, it’s often easier to troubleshoot issues by identifying the port directly instead of using the service name.

Show connections on IPv6 ports:

lsof -i6

Show connections on IPv6 ports

Compared to the lsof -i command earlier, this command specifically looks for services with a TYPE of IPv6.

Show connections on a specific port without resolving service name:

lsof -Pi :25

Show connections on a specific port without resolving service name

Looking specifically for all current processes listening on port 25 shows that there are two current processes on the system (one IPv4, one IPv6) running. Neither of these processes is listening for external connections.

Show listening ports:

lsof -i -sTCP:LISTEN

Show listening ports

This command lists all running processes that are in the LISTEN state, meaning they are actively listening for incoming connections.

Show who’s using a file:

lsof /path/of/file

You can see from the above command two processes are actively running and using the /var/log/messages file in some way.

Recursively show all open files in a directory:

lsof +D /usr/lib

Recursively show all open files in a directory

On our test system, using the +D flag shows six separate open processes originating from the /usr/lib directory.

Show open files by user:

lsof -u atlantic-noc

Show open files by user

The atlantic-noc user has an active SSH session with a process id (PID) of 7769.

Show all open files with specific process ID:

lsof -p 7790

Show all open files with specific process ID

The -p option filtered out open files so only the ones with the specified process id are showed. In this case, PID 7790. You can select multiple PIDs by specifying a comma, such as -p 7789,7790

Show all network activity for a specific user:

lsof -a -u atlantic-noc -i

The -a flag combines (changes default OR behavior to AND) the -u and -i flags to provide a list of network file usage for user atlantic-noc; in this case only a single SSH session.

Anything Else?

The commands presented above are just the beginning for lsof; a versatile tool that can be used either by itself or in combination with other Unix troubleshooting tools, such as strace. But that’s an article for another time.

Remember to check our blog for any updates to this article, other Linux related posts, and further Unix command primers or learn more about our reliable VPS hosting services.