Verified and Tested 10/30/15
Introduction
At some point in their career, everyone is going to troubleshoot an issue. And everyone has their favorite troubleshooting step to use. For a lot of system admins, the first step to do is make sure that everything is up-to-date. For others it’s only a matter of checking log files before finding your solution. For me and others here at Atlantic.Net, it’s lsof; a command line utility used to list information about files that are opened by various processes.
Mostly used for troubleshooting network connection issues, lsof is a powerful yet too-little-known application. If you’re familiar with general Linux commands, lsof is an easy tool to remember because it “lists open files”.
Prerequisites
lsof is a Unix-specific command. In order to use it for identifying system information you will need to be using a Unix-style system, such as CentOS or Ubuntu. lsof comes pre-installed on these systems.
If you are using a Windows system, you can get similar results by using the Process Explorer menu, as well as both the netstat and sysinternals commands.
lsof Usage Flags
One thing that’s important to understand is that lsof has a large number of available flags that can be used to modify the output you receive. Below shows all of the available flags:
[[email protected] ~]# lsof --help usage: [-?abhlnNoOPRtUvVX] [+|-c c] [+|-d s] [+D D] [+|-f[gG]] [+|-e s] [-F [f]] [-g [s]] [-i [i]] [+|-L [l]] [+m [m]] [+|-M] [-o [o]] [-p s] [+|-r [t]] [-s [p:s]] [-S [t]] [-T [t]] [-u s] [+|-w] [-x [fl]] [--] [names]
Going through each flag would be a much longer article, so we’ll keep things simple and cover only the most used day-to-day options.
Key Option Flags
It’s important to understand how lsof works, especially by default. When you are passing options to the lsof command the default behavior is to OR the results. This means that running lsof using both the -i as well as the -p flag you will get the results of both.
These are the most often used flags in our troubleshooting:
default : with no options flags, lsof lists all open files for active processes grouping : it's possible to group options, such as passing the flags though -nPi. Just keep an eye out for which options take additional parameters -a : AND the result (instead of the default OR) -l : show the userID instead of the username in the output -h : get help -i : selects the listing of files that match -n : stop host name resolution of network numbers -t : get process IDs only -P : stop conversion of port numbers to port names
Getting Started
List all open files.
# lsof
Show all connections:
# lsof -i
The server we’re testing on is a new Atlantic.Net CentOS6 server which does not have any additional software packages installed. You can see from the above output that the server is only listening for SSH and SMTP traffic. If you want to see the actual port number instead of the service name we will need to run the command using the -Pi flag:
lsof -Pi
As you can see, all of the *:ssh and localhost:smtp output has been converted to *:22 and localhost:25. For most system administrators, especially if you’re not used to going through and analyzing system services, it’s often easier to troubleshoot issues by identifying the port directly instead of using the service name.
Show connections on IPv6 ports:
lsof -i6
Compared to the lsof -i command earlier, this command specifically looks for services with a TYPE of IPv6.
Show connections on a specific port without resolving service name:
lsof -Pi :25
Looking specifically for all current processes listening on port 25 shows that there are two current processes on the system (one IPv4, one IPv6) running. Neither of these processes is listening for external connections.
Show listening ports:
lsof -i -sTCP:LISTEN
This command lists all running processes that are in the LISTEN state, meaning they are actively listening for incoming connections.
Show who’s using a file:
lsof /path/of/file
You can see from the above command two processes are actively running and using the /var/log/messages file in some way.
Recursively show all open files in a directory:
lsof +D /usr/lib
On our test system, using the +D flag shows six separate open processes originating from the /usr/lib directory.
Show open files by user:
lsof -u atlantic-noc
The atlantic-noc user has an active SSH session with a process id (PID) of 7769.
Show all open files with specific process ID:
lsof -p 7790
The -p option filtered out open files so only the ones with the specified process id are showed. In this case, PID 7790. You can select multiple PIDs by specifying a comma, such as -p 7789,7790
Show all network activity for a specific user:
lsof -a -u atlantic-noc -i
The -a flag combines (changes default OR behavior to AND) the -u and -i flags to provide a list of network file usage for user atlantic-noc; in this case only a single SSH session.
Anything Else?
The commands presented above are just the beginning for lsof; a versatile tool that can be used either by itself or in combination with other Unix troubleshooting tools, such as strace. But that’s an article for another time.
Remember to check our blog for any updates to this article, other Linux related posts and further Unix command primers or learn more about our reliable VPS hosting services.
