Thursday, October 30, 2014

A hitchhikers guide to troubleshooting linux memory usage

A hitchhikers guide to troubleshooting linux memory usage
Linux memory management has always intrigued me. While learning Linux, many concepts are confusing at first, but after a lot of reading, googling, understanding and determination, I learned that the kernel is not only efficient at memory
management but also on par with artificial intelligence in making memory distribution decisions..
This post will hopefully show you how to troubleshoot or at least find out the
amount of memory used by Linux and an application running on it. If you have any
doubts, do let me know by commenting.
Finding Linux System Memory usage

One of the simplest way to check Linux system memory usage is with the “free”
Below is my “free -­m” command output.
linux memory usage

The first line shows you that my free memory is only 111MB but the trick here is to
look at the second line for free memory.
The first line calculates caches and buffers along with the used memory.
Now linux does cache data to speed up the process of loading content.
But, that cached memory is also available for a new process to use at any time and
can be freed by the kernel immediately, in case any of your processes need it.
Buffers on the other hand, store metadata like file permissions or memory location of the cached data. Since this physical memory is availble for our process to use, we can subtract this information from the used memory to give us a free memory of 305MB as seen in the figure above.
Memory caching or Page cache
Linux divides memory into blocks called pages and hence the term page cache.
I will be using page cache from now on, but don’t get confused just replace page with memory if you do.
How page cache works.
Any time you do a read() from a file on disk, that data is read into memory, and
goes into the page cache. After this read() completes, the kernel has the option to
discard the page, since it is not being used. However, if you do a second read of
the same area in a file, the data will be read directly out of memory and no trip to
the disk will be taken. This is an incredible speedup.  And is the reason why Linux
uses its page cache so extensively, is because it knows that once you access a
page on disk the first time, you will surely access it again.
Similarly when you save data to a file it is not immediately written to the disk, it is
cached and written periodically to reduce I/O. The name for this type of cache is
Dirty.You can see it’s output  by running “cat /proc/meminfo”.
linux memory usage
You can flush the cache with the following command.
echo 1 > /proc/sys/vm/drop_caches
To write cache to disk you can use the sync command
Finding linux process memory usage 
Here is my HTOP output.
linux memory usage
You need to look at the VIRT, RSS and SHR columns to get an idea of memory

VIRT : Stands for Virtual Memory and displays the amount of memory requested
by an application. Applications often request more memory than required, however
they may not be actually using that memory and hence we can ignore this column.
RSS : Stands for Resident Set Size and displays the amount of memory used by
the process.
SHR : Stands for  Shared memory and displays the memory shared with other
The last two columns are what we need look at  to find out how much memory our
process is using.
For simple linux applications this information should suffice for you to know which
process is taking too much of your memory. But if you need to debug advance
issues like a memory leak then you need to go a step further.

The only problem with the HTOP output is that the RSS column displays used memory as Process memory + Total shared memory, even though the process is
using only a part of the shared memory.

Let’s take an analogy to understand this better.
I am a sensible spender ( I am married :) ), so sometimes I like to carpool to work.
Let’s say it takes 4$ worth of fuel from home to office.
When I go to work alone, I spend 4$ on fuel. The next day I car pool with 3 of my
friends, we pay a dollar each on fuel. So my total expenditure for the two days
would be 5$, however RSS would display it as $8.
Therefore, in order to find the exact memory usage you can you use a tool called
git clone

cd ps_mem

sudo ./

linux memory usage
There you go php­fpm is hogging my memory.
Troubleshooting slow application issues in Linux.
If you look at the free output again you will see that the swap memory is used even
though we have ram free
linux memory usage
The Linux kernel moves out pages which are not active or being used at the
moment to swap space on the disk. This process is known as
swappiness. Since swap space is on the hard drive fetching
data will be slower as compared to your ram, This may cause your application to take a hit in
terms of speed. You have the option to turn off swaping by changing the value in
 “/proc/sys/vm/swappiness” to 0. The value ranges from 0 to 100 where 100
means aggressive swapping.
Update: A good tip from karthik in comments
“I would recommend 1 more step, before changing the swappiness value. Try “vmstat -n 1″ and check the “si”, “so” field. If “si” and “so” (stands for swapin and swapout) fields are always 0, then the system is currently not swapping. Some application, has used the swap but somehow its not cleaned the swap space. At such situation a “swapoff/swapon” command would be handy.”
Update 2: Another good tool and page cache advice from reddit user zeroshiftsl
“I would add one more section though, the slab. I recently ran into an issue where a system was consuming more and more memory over time. I thought it was a leak, but no process seemed to own any of the missing memory. Htop showed the memory as allocated but it didn’t add up in the processes. This was NOT disk cached memory. Using “slabtop“, I found that a bunch of memory was stuck in dentry and inode_cache. This memory was not being freed when dropping caches like it should, and upping the vfs_cache_pressure had no effect. Had to kill the parent process (SSH session) that created all of these to reclaim the memory.”
Update: The script runs only once, you may want to run it periodically to get real time memory usage, hence I recommend you read How to display a changing output like top
I tried to keep this post as simple as possible and this data should give you enough
information to troubleshoot any memory usage issues you might face on your linux
vps or server.
If there is anything I missed please do share your experiences troubleshooting linux memory usage issues in the comments below.

How to create and manage LXC containers on Ubuntu

While the concept of containers was introduced more than a decade ago to manage shared hosting environments securely (e.g., FreeBSD jails), Linux containers such as LXC or Docker have gone mainstream only recently with the rising need to deploy applications for the cloud. While Docker is getting all the media spotlight these days with strong backing from major cloud providers (e.g., Amazon AWS, Microsoft Azure) and distro providers (e.g., Red Hat, Ubuntu), LXC is in fact the original container technology developed for Linux platforms.
If you are an average Linux user, what good does Docker/LXC bring to you? Well, containers are actually a great means to switch between distros literally instantly. Suppose your current desktop is Debian. You want Debian's stability. At the same time, you also want to play the latest Ubuntu games. Then instead of bothering to dual boot into a Ubuntu partition, or boot up a heavyweight Ubuntu VM, simply spin off a Ubuntu container on the spot, and you are done.
Even without all the goodies of Docker, what I like about LXC containers is the fact that LXC can be managed by libvirt interface, which is not the case for Docker. If you have been using libvirt-based management tools (e.g., virt-manager or virsh), you can use those same tools to manage LXC containers.
In this tutorial, I focus on the command-line usage of standard LXC container tools, and demonstrate how to create and manage LXC containers from the command line on Ubuntu.

Install LXC on Ubuntu

To use LXC on Ubuntu, install LXC user-space tools as follows.
$ sudo apt-get install lxc
After that, check the current Linux kernel for LXC support by running lxc-checkconifg tool. If everything is enabled, kernel's LXC support is ready.
$ lxc-checkconfig

After installing LXC tools, you will find that an LXC's default bridge interface (lxcbr0) is automatically created (as configured in /etc/lxc/default.conf).

When you create an LXC container, the container's interface will automatically be attached to this bridge, so the container can communicate with the world.

Create an LXC Container

To be able to create an LXC container of a particular target environment (e.g., Debian Wheezy 64bit), you need a corresponding LXC template. Fortunately, LXC user space tools on Ubuntu come with a collection of ready-made LXC templates. You can find available LXC templates in /usr/share/lxc/templates directory.
$ ls /usr/share/lxc/templates

An LXC template is nothing more than a script which builds a container for a particular Linux environment. When you create an LXC container, you need to use one of these templates.
To create a Ubuntu container, for example, use the following command-line:
$ sudo lxc-create -n -t ubuntu

By default, it will create a minimal Ubuntu install of the same release version and architecture as the local host, in this case Saucy Salamander (13.10) 64-bit.
If you want, you can create Ubuntu containers of any arbitrary version by passing the release parameter. For example, to create a Ubuntu 14.10 container:
$ sudo lxc-create -n -t ubuntu -- --release utopic
It will download and validate all the packages needed by a target container environment. The whole process can take a couple of minutes or more depending on the type of container. So be patient.

After a series of package downloads and validation, an LXC container image are finally created, and you will see a default login credential to use. The container is stored in /var/lib/lxc/. Its root filesystem is found in /var/lib/lxc//rootfs.
All the packages downloaded during LXC creation get cached in /var/cache/lxc, so that creating additional containers with the same LXC template will take no time.
Let's see a list of LXC containers on the host:
$ sudo lxc-ls --fancy
test-lxc   STOPPED  -     -     NO         
To boot up a container, use the command below. The "-d" option launches the container as a daemon. Without this option, you will directly be attached to console right after you launch the container.
$ sudo lxc-start -n -d
After launching the container, let's check the state of the container again:
$ sudo lxc-ls --fancy
lxc   RUNNING  -     NO         
You will see that the container is in "RUNNING" state with an IP address assigned to it.
You can also verify that the container's interface (e.g., vethJ06SFL) is automatically attached to LXC's internal bridge (lxcbr0) as follows.
$ brctl show lxcbr0

Manage an LXC Container

Now that we know how to create and start an LXC container, let's see what we can do with a running container.
First of all, we want to access the container's console. For this, type this command:
$ sudo lxc-console -n

Type to exit the console.
To stop and destroy a container:
$ sudo lxc-stop -n
$ sudo lxc-destroy -n
To clone an existing container to another, use these commands:
$ sudo lxc-stop -n
$ sudo lxc-clone -o -n


For those of you who encounter errors with LXC, here are some troubleshooting tips.
1. You fail to create an LXC container with the following error.
$ sudo lxc-create -n test-lxc -t ubuntu
lxc-create: symbol lookup error: /usr/lib/x86_64-linux-gnu/ undefined symbol: cgmanager_get_pid_cgroup_abs_sync
This means that you are running the latest LXC, but with an older libcgmanager. To fix this problem, you need to update libcgmanager.
$ sudo apt-get install libcgmanager0

Wednesday, October 29, 2014

How to encrypt files and directories with eCryptFS on Linux

You do not have to be a criminal or work for the CIA to use encryption. You simply don't want anybody to spy on your financial data, family pictures, unpublished manuscripts, or secret notes where you have jotted down startup ideas which you think can make you super rich.
I have heard people telling me "I'm not important enough to be spied on" or "I don't hide anything to care about." Well, my opinion is that even if I don't have anything to hide, or I can publish a picture of my kids with my dog, I have the right to not do it and want to protect my privacy.

Types of Encryption

We have largely two different ways to encrypt files and directories. One method is filesystem-level encryption, where only certain files or directories (e.g., /home/alice) are encrypted selectively. To me, this is a perfect way to start. You don't need to re-install everything to enable or test encryption. Filesystem-level encryption has some disadvantages, though. For example, many modern applications cache (part of) files in unencrypted portions of your hard drive, such as swap partition, /tmp and /var folders, which can result in privacy leaks.
The other way is so-called full-disk encryption, which means that the entire disk is encrypted (possibly except for a master boot record). Full disk encryption works at the physical disk level; every bit written to the disk is encrypted, and anything read from the disk is automatically decrypted on the fly. This will prevent any potential unauthorized access to unencrypted data, and ensure that everything in the entire filesystem is encrypted, including swap partition or any temporarily cached data.

Available Encryption Tools

There are several options to implement encryption in Linux. In this tutorial, I am going to describe one option: eCryptFS a user-space cryptographic filesystem tool. For your reference, here is a roundup of available Linux encryption tools.

Filesystem-level encryption

  • EncFS: one of the easiest ways to try encryption. EncFS works as a FUSE-based pseudo filesystem, so you just create an encrypted folder and mount it to a folder to work with.
  • eCryptFS: a POSIX compliant cryptographic filesystem, eCryptFS works in the same way as EncFS, so you have to mount it.

Full-disk encryption

  • Loop-AES: the oldest disk encryption method. It is really fast and works on old system (e.g., kernel 2.0 branch).
  • DMCrypt: the most common disk encryption scheme supported by the modern Linux kernel.
  • CipherShed: an open-source fork of the discontinued TrueCrypt disk encryption program.

Basics of eCryptFS

eCryptFS is a FUSE-based user-space cryptographic filesystem, which has been available in the Linux kernel since 2.6.19 (as ecryptfs module). An eCryptFS-encrypted pseudo filesystem is mounted on top of your current filesystem. It works perfectly on EXT filesystem family and others like JFS, XFS, ReiserFS, Btrfs, even NFS/CIFS shares. Ubuntu uses eCryptFS as its default method to encrypt home directory, and so does ChromeOS. Underneath it, eCryptFS uses AES algorithm by default, but it supports others algorithms, such as blowfish, des3, cast5, cast6. You will be able to choose among them in case you create a manual setup of eCryptFS.
Like I said, Ubuntu lets us choose whether to encrypt our /home directory during installation. Well, this is the easiest way to use eCryptFS.

Ubuntu provides a set of user-friendly tools that make our life easier with eCryptFS, but enabling eCryptFS during Ubuntu installation only creates a specific pre-configured setup. So in case the default setup doesn't fit your needs, you will need to perform a manual setup. In this tutorial, I will describe how to set up eCryptFS manually on major Linux distros.

Installation of eCryptFS

Debian, Ubuntu or its derivatives:
$ sudo apt-get install ecryptfs-utils
Note that if you chose to encrypt your home directory during Ubuntu installation, eCryptFS should be already installed.
CentOS, RHEL or Fedora:
# yum install ecryptfs-utils
Arch Linux:
$ sudo pacman -S ecryptfs-utils
After installing the package, it is a good practice to load the eCryptFS kernel module just to be sure:
$ sudo modprobe ecryptfs

Configure eCryptFS

Now let's start encrypting some directory by running eCryptFS configuration tool:
$ ecryptfs-setup-private

It will ask for a login passphrase and a mount passphrase. The login passphrase is the same as your normal login password. The mount passphrase is used to derive a file encryption master key. Leave it blank to generate one as it's safer. Log out and log back in.
You will notice that eCryptFS created two directories by default: Private and .Private in your home directory. The ~/.Private directory contains encrypted data, while you can access corresponding decrypted data in the ~/Private directory. At the time you log in, the ~/.Private directory is automatically decrypted and mapped to the ~/Private directory, so you can access it. When you log out, the ~/Private directory is automatically unmounted and the content in the ~/Private directory is encrypted back into the ~/.Private directory.
The way eCryptFS knows that you own the ~/.Private directory, and automatically decrypts it into the ~/Private directory without needing us to type a password is through an eCryptFS PAM module which does the trick for us.
In case you don't want to have the ~/Private directory automatically mounted upon login, just add the "--noautomount" option when running ecryptfs-setup-private tool. Similarly, if you do not want the ~/Private directory to be automatically unmounted after logout, specify "--noautoumount" option. But then, you will have to mount or unmount ~/Private directory manually by yourself:
$ ecryptfs-mount-private ~/.Private ~/Private
$ ecryptfs-umount-private ~/Private
You can verify that .Private folder is mounted by running:
$ mount

Now we can start putting any sensitive files in ~/Private folder, and they will automatically be encrypted and locked down in ~/.Private folder when we log out.
All this seems pretty magical. Basically ecryptfs-setup-private tool makes everything easy to set up. If you want to play a little more and set up specific aspects of eCryptFS, go to the official documentation.


To conclude, if you care a great deal about your privacy, the best setup I recommend is to combine eCryptFS-based filesystem-level encryption with full-disk encryption. Always remember though, file encryption alone does not guarantee your privacy.