Lighweight virtualization with LXC

LXC is lightweight virtualization system for Linux. As its homepage states, its a "chroot on steroids", which is great if you want to spin off a bunch of processes in an isolated environment without an overhead of running a complete virtualization like VirtualBox or VMWare. LXC also provides a set of mechanisms to manage guest OS resources like RAM and CPU usage.

My interest in this project started with a search for a simple solution to isolate multiple processes on the system without big customizations to the host OS. Plus, i figured out that heroku were using LXC for their cloud deployment platform so decided to give it a shot. Overall experience was quite positive, although i stumbled upon a few minor issues, mostly due to host OS differences and configuration mistakes. I'd say it works quite well for what it was designed for. As a result of my experiments i've got a bunch of scripts, tools and even side projects that i use every day.

Unfortunately, there is no way to run LXC on Mac OSX (as far as i know) and if you dont have a server with root access around you've got only one option: VirtualBox (or VMWare). VirtualBox is pretty easy to manage, but take a look at vagrant - set of automation tools to manage virtual machines with ease. With vagrant you'll get a fresh VM up and running in a matter of minutes.

Getting Started

Make sure you have VirtualBox and vagrant gem installed on your local system. Then, create a folder for your new project, i usually store vm-projects under ~/Vagrant/.

Initialize a new vagrant vm:

vagrant init

That'll create a new Vagrantfile config. Edit with the following:

Vagrant::Config.run do |config|
  config.vm.box = "precise64"
  config.vm.box_url = "http://files.vagrantup.com/precise64.box"
  config.vm.network :bridged, :bridge => "en1: Wi-Fi (AirPort)"
  config.vm.share_folder "work", "/root/shared", "./shared"
end

Configuration above will use Ubuntu 12.04 64bit installation, one of the official vagrant images. Network configuration is set to bridged in order to access virtual machine directly via ssh. On OSX it'll map to wifi interface automatically without a prompt. Also, for all vm-related experiments there is a shared folder where you can place all project scripts.

The next step would be getting the VM up:

vagrant up

For the first time it'll take around 10 minutes (depends on your internet connection) and as a result you'll have a new VM ready to roll.

Configure environment

Once you have a new virtual machine ready, we'll need to download a few packages and scripts to prepare and configure LXC environment.

vagrant ssh
sudo su

Execute the configuration script (bash):

apt-get update -y
apt-get install -y git-core ruby1.9.3 rake
apt-get install -y lxc squashfs-tools

We need squashfs filesystem tools to package containers' root filesystems into an image file. Ruby is needed for general scripting only, any version >= 1.8.7 is fine.

Now, we need some tools to experiment with LXC:

gem install mustache terminal_helpers
git clone git://github.com/sosedoff/lxc-tools.git
cd lxc_tools && rake install

I created this set of minimalistic tools to get up and running with LXC very quickly without much of configuration or dependencies. After you install everything, you'll have 3 major scripts installed into your system:

lxc-setup-rootfs - To create a bootstrapped OS image
lxc-setup-container - To create a new LXC container
lxc-setup-network - To configure network interface for LXC. Not needed on Ubuntu 12.04

Host OS should be good to go.

Create container

First thing we need to do before we can create a new container is to bootstrap a rootfs. The following command will extract everything under '~/image` dir:

lxc-setup-rootfs --path ~/image --release 12.04

Next, create a directory where all new containers will be stored:

mkdir ~/containers

Create a new container:

lxc-setup-container \
  --name box
  --path ~/containers/box
  --rootfs ~/image
  --net-ip 10.0.3.2
  --net-gateway 10.0.3.1

Container will have a root user with default password root. Now, start it in background mode:

lxc-start -n box -d

Check, if its running:

lxc-ls # Should print 2 lines of "box"
ping -c 5 10.0.3.2

Connect via SSH:

ssh 10.0.3.2 -l root

You are now logged into an isolated environment that is no different from a regular linux server. Although, you may notice that it wont have any basic packages, thats due to original debootstrap process, which installs only a fix amount of initial packages and leaving that for you to finish and customize.

Use container images

Another great use of LXC would be spinning up a new container from a prebuilt image in a few seconds. You can build images with tools of your choice: Chef-Solo, Sprinkle or bash scripts.

For packaging container images you could use a Squashfs, a compressed read-only filesystem and intended for general read-only filesystem use, for archival use (in cases where a .tar.gz file may be used), and in constrained block device/memory systems (embedded systems) where low overhead is needed. The tradeoff is that you can simply mount it or extract, where images packed into tarball could only be extracted.

Make an image (dont forget to stop container):

lxc-stop -n box
mksquashfs ~/containers/box/rootfs ~/box.sqsh

That will produce a compressed rootfs image of your container that you can reuse later, distribute over the network or upload to cloud storage for better availability.

Extract the image:

unsquashfs -d ~/extract/to/path ~/box.sqsh

Or simply mount it:

mkdir -p ~/boxes/box
mount -t squashfs ~/box.sqsh ~/boxes/box

Then you can pass extracted or mounted path to lxc-setup-container as --rootfs parameter.

Summary

LXC is a great tool and you should definitely try it. It has been used in production by many cloud deployment providers. Personally i have a set of applications with lxc in use, such as CI system, deployment platform for staging rails and wordpress applications. It could be a very powerful tool in your infrastructure at a minimal cost.