When Bad Things Start to Happen (Part II – Linux)

We have previously blogged about what information to collect from a Windows box when we suspect that it has been compromised. Now, we will discuss what information we can get from a Linux system without launching a full-blown forensic investigation.

Generally, the logic is the same - we will be searching for the attack vector; modifications on the system, resulting in persistence;  unusual processes and commands; and changes in the system environment. We would still need to get the information from the network level as described in the previous post.

Again the entire process can be scripted; I have used Python to create a script that collects all the data from several Linux distributions and HP-UX and produces a password-protected archive with all the information in it, but any scripting language can be used.

Getting the Users

We should start with collecting the current users on the system, their shells, groups, and home directories. This will achieve three important things:

  • we will spot any new or unusual users that do not follow the naming convention;
  • we will have the users that are added in groups where they do not belong;
  • and we will have their $HOMEs, so that we can collect history and configuration files.

We will also spot users with login shells who are not supposed to be able to log in, as well as users with unusual shells.

When you have the users, you can proceed with

Getting History Files

This should be pretty straightforward. Just go through the user $HOMEs and collect the history files. The history files depend on the users’ shells; furthermore, attackers may start a different shell after they log in, so you may collect *history files.

Usually, one of the first things an attacker would do is disable history by setting the $HISTFILE to /dev/null, but in practice I’ve seen that this is not always the case. Even in this case, you may still find valuable traces, e.g. the command that disables the history would be logged after the user terminates the shell, and may still be there. This depends on whether the attacker is a human or a bot, how skillful they are, and so on.

If the system has system accounting, we may also get the output of `lastcomm`. This will get us information about previously executed commands.

We also need to collect the currently

Logged-on Users

We can do this with the `w` command. It will show the username, the terminal the IP address of the user, when they logged in, their idle time and what they are doing.

More detailed information about what is going on currently on the system can be obtained by collecting

The Running Processes

On Linux, we can use the `ps auxwwwwf` command to do that. It will produce a nice tree of the currently running processes on the system.

Since the user can be long gone, we should also get the information about the

Login Attempts

using the `last` and `lastb` commands. They will give us information about the last good and bad login attempts. To cover a larger period of time, we may also run the commands with the ‘-f’ switch against older files.

Of course, we will also have to collect

System Information

We usually collect the following information about the box:

  • the platform architecture;
  • the Linux distribution;
  • the libc version;
  • the hostname;
  • the kernel version;
  • the Java version;

This information will give us an idea about what we are dealing with, so that we can narrow down our investugation.

For this purpose, we would also need to collect the

Installed Packages

Now, how to do this depends on the distribution and the package manager you are using. If it is dpkg, then `dpkg -l` will list all packages with their versions. If you are an RPM fan, than use `rpm -qi`. This will help you detect outdated and potentially vulnerable packages.

One of the most crucial piece of data we have to collect is the

Network Information

There are several things we need:

  • the output of the `netstat -anp` command;
  • the ‘hosts’ file;
  • the output of `route -n`;
  • the ‘resolv.conf’ file;
  • the ARP cache.

This will provide us with information about established connections, listening sockets and modifications of the configuration.

In that context, we would also have to collect data about the

Startup Services

There are several ways to do that, depending on the distribution. If using Upstart, you can use `service –status-all`. RHEL has the `chkconfig –list` command. But probably the best way is still to walk through the /etc/rc*.d directories and get the file names for each level.

Another important piece of data that should be collected is the

Host Firewall and IDS/IPS Information

If you are using a firewall, the attacker may have modified it to allow access for themselves. So, the output of `iptables -L -n` is must have. If there is a host IDS/IPS, its logs should be collected too.

And while we are on that, we can as well get all the

System Logs

from the machine under investigation. I would usually collect the entire ‘/var/log’ directory. If there are utilities that log elsewhere, their log files should be collected as well.

This is where it would be most likely to find information about the attack vector.

Other useful data we need to get is the

Crontab Files

We should get the /etc/crontab file, the files in /etc/cron.*, and the user crontab files from /var/spool/cron. We have seen enough cases, in which malicious processes were being started from cron jobs, so it is essential to check these.

It is also advised to get a list of the

Open Files

On Linux, we will use the `lsof` command to do that. It may take a while to complete, but it will show the files opened by the currently running processes.

Finally, we should get the loaded

Kernel Modules

Sometimes attackers may load a kernel module, and if this is the case, we definitely need to know that. We can use the `lsmod` command to do this.


we should say that this is not a complete list by any means.  Get a good understanding of the system and act accordingly.

Leave a Reply