Verified and Tested 10/30/15
At some point in their career, everyone is going to troubleshoot an issue. And everyone has their favorite troubleshooting step to use. For a lot of system admins, the first step to do is make sure that everything is up-to-date. For others it’s only a matter of checking log files before finding your solution. For me and others here at Atlantic.Net, it’s lsof; a command line utility used to list information about files that are opened by various processes.
Mostly used for troubleshooting network connection issues, lsof is a powerful yet too-little-known application. If you’re familiar with general Linux commands, lsof is an easy tool to remember because it “lists open files”.
lsof Usage Flags
One thing that’s important to understand is that lsof has a large number of available flags that can be used to modify the output you receive. Below shows all of the available flags:
[root@OffsiteLinux ~]# lsof --help usage: [-?abhlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-f[gG]] [+|-e s] [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s] [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Going through each flag would be a much longer article, so we’ll keep things simple and cover only the most used day-to-day options.
Key Option Flags
It’s important to understand how lsof works, especially by default. When you are passing options to the lsof command the default behavior is to OR the results. This means that running lsof using both the
-i as well as the
-p flag you will get the results of both.
These are the most often used flags in our troubleshooting:
default : with no options flags, lsof lists all open files for active processes grouping : it's possible to group options, such as passing the flags though -nPi. Just keep an eye out for which options take additional parameters -a : AND the result (instead of the default OR) -l : show the userID instead of the username in the output -h : get help -i : selects the listing of files that match -n : stop host name resolution of network numbers -t : get process IDs only -P : stop conversion of port numbers to port names
List all open files.
Show all connections:
# lsof -i
The server we’re testing on is a new Atlantic.Net CentOS6 server which does not have any additional software packages installed. You can see from the above output that the server is only listening for SSH and SMTP traffic. If you want to see the actual port number instead of the service name we will need to run the command using the
As you can see, all of the *:ssh and localhost:smtp output has been converted to *:22 and localhost:25. For most system administrators, especially if you’re not used to going through and analyzing system services, it’s often easier to troubleshoot issues by identifying the port directly instead of using the service name.
Show connections on IPv6 ports:
Compared to the
lsof -i command earlier, this command specifically looks for services with a TYPE of IPv6.
Show connections on a specific port without resolving service name:
lsof -Pi :25
Looking specifically for all current processes listening on port 25 shows that there are two current processes on the system (one IPv4, one IPv6) running. Neither of these processes is listening for external connections.
Show listening ports:
lsof -i -sTCP:LISTEN
This command lists all running processes that are in the LISTEN state, meaning they are actively listening for incoming connections.
Show who’s using a file:
Recursively show all open files in a directory:
lsof +D /usr/lib
On our test system, using the
+D flag shows six separate open processes originating from the /usr/lib directory.
Show open files by user:
lsof -u atlantic-noc
The atlantic-noc user has an active SSH session with a process id (PID) of 7769.
Show all open files with specific process ID:
lsof -p 7790
-p option filtered out open files so only the ones with the specified process id are showed. In this case, PID 7790. You can select multiple PIDs by specifying a comma, such as
Show all network activity for a specific user:
lsof -a -u atlantic-noc -i
The commands presented above are just the beginning for lsof; a versatile tool that can be used either by itself or in combination with other Unix troubleshooting tools, such as strace. But that’s an article for another time.