Rhonda's Blog                    
Mainpage Disclaimer

Wed, 23 Mar 2011

LXC and NAT on notebook

Yesterday I was hinted towards lxc when I wondered what happened to openvz in unstable (which unfortunately isn't documented at all in the kernel changelogs, but that's a different story). So I started off taking a look. From a bit of experimenting around with it I consider it something that I want to play more with, and I want to share the problems I stumbled upon with you so that you don't have to figure them out on your own.

First of all, LXC uses the cgroup kernel facility for resource management. The according file system isn't mounted by default, and LXC doesn't care for where it is mounted, it just needs to be. It seems like /sys/fs/cgroup seems to be the proper place (see 601757), so add the line cgroup /sys/fs/cgroup cgroup defaults 0 0 to your /etc/fstab file and sudo mount cgroup it.

Next, it seems like bridging is the defacto standard for networking with lxc, but given that I want to use it on my notebook while being mobile I can't bind the bridge to any specific interface. To make this happen, one needs the bridge-utils package installed, and secondly, this is the path that I chose. I've added to /etc/network/interfaces this snippet:

auto br0
iface br0 inet static
    bridge_maxwait 0
    bridge_ports dummy0

This will bring up the bridge and act as gateway. For the running system, call sudo ifup br0. To make the host universally being able to work as gateway, of course ip_forward needs to be enabled. For this I added the line net.ipv4.ip_forward=1 to /etc/sysctl.d/local.conf (and for the running system, echo 1 into /proc/sys/net/ipv4/ip_forward).

As I am using ferm for configuring the firewall on my notebook I have to add some parts into its configuration. This is the raw part that needs to get added, mix it into your existing configuration:

table filter {
    chain INPUT {
        # allow DNS queries from LXContainers
        proto (udp tcp) dport domain source ACCEPT;

    chain FORWARD {
        # allow LXContainers into the net
        source ACCEPT;

table nat {
    chain POSTROUTING {
        # NAT LXContainers
        source MASQUERADE;

For DNS I installed dnsmasq so that I won't have to touch the /etc/resolv.conf inside the containers whenever I switch networks.

So far for the host part, now to the actual containers. There is the /usr/lib/lxc/templates/lxc-debian helper script which uses debootstrap to create you a lenny chroot—at least in the squeeze package this is hardwired, likewise with using cdn.debian.net. Copy the script and edit it to your likes if you feel like it. From what I understood it expects you to store the containers below /var/lib/lxc, I haven't yet tested for different places. So this was my commandline for that:
sudo /usr/lib/lxc/templates/lxc-debian -p /var/lib/lxc/vm0

A while later you'll end up below that directory with two entries: The config file and the rootfs subdirectory which is actually the bootstrapped distribution part.

Now comes the configuration of the container. Open the config file with your favorite editor and add the following lines to the end:

lxc.utsname = vm0
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
# lxc.network.name = eth0
lxc.network.hwaddr = 00:FF:80:80:80:80
lxc.network.ipv4 =

The network.name part is commented out, it defaults to that name internally; you though can change it to whatever you prefer. Caution, even though this is the documented approach, it does not work for Debian containers. It will always try to get its IP address through dhcp, lxc.network.ipv4 has no meaning for us. We need to change inside the rootfs the file etc/network/interfaces to read like this instead:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static

I suggest to keep the config and the interfaces file aligned with respect to the ipv4 setting so if this gets fixed upstream you won't stumble into any surprises. Also like mentioned before, we need to change the nameserver entries inside the rootfs file etc/resolv.conf to read nameserver

Now it's time to start it up and log in! sudo lxc-start -n vm0 -d will start the container in the background, and sudo lxc-console -n vm0 will give you the login to the container. The default password for the root user is root, obviously you want to change that before you install any networking services into the container like ssh-server. In case you want to quit from that console notice the message upon starting it, it's bound to <Ctrl+a q>.

One more issue that I had: The default route wasn't set. I had to manually call ip r a default via dev eth0 to be able to use the network inside the container. It seems to be related to that netbase isn't installed by default. If you install it the default route will be set upon starting the container automatically.

This should get you started, there is of course more to explore and experiment with. Actually it is also suggested to create a tarball from your vm0 after you did the basic setup and installed the basic components you want to have around so you won't have to bootstrap over and over again. Do this after you have shut down the container, either through a halt from a container shell or through sudo lxc-stop -n vm0. The tarball can then get extracted to a different directory and just needs minor tweaks in the config and rootfs/etc/network/interfaces file to not create any clash with other containers (lxc.rootfs, lxc.mount.entry, lxc.utsname, lxc.network.hwaddr and ipv4 address).

About limiting the containers, you can do it dynamically through the cgroup file system, and set it permanently through the config file. See man lxc.conf about these settings, amongst others.

Enjoy, use, experiment. With sudo lxc-checkconfig you will see what your kernel actually supports for your LXCs. You will most probably notice the missing for the memory controller, this is tracked in the Debian bug report 534964.

/debian | permanent link | Comments: 6

Trackbacks are closed for this story.

smcv wrote at 2011-03-23 17:15:

cgroups are canonically mounted at /sys/fs/cgroup (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=601757).

Rhonda wrote at 2011-03-23 17:41:

Thanks for the notice! I'll adjust the blog article to it. :)

novalix wrote at 2011-03-23 17:50:

The path to the containers seems to be hard wired to /var/lib/lxc in the helper programs like lxc-start etc. I haven 't taken a look in the source code of all of them, but if you change the destination path in the lxc-create script (which is easy because it is a simple shell script), it will create the container in the specified path to no avail, since lxc-start will complain about the missing container.

Btw, Nice writeup

Konstantin Khomoutov wrote at 2011-03-24 07:51:

Note that lxc-stop does not do a sensible thing with regard to a "full" system (a system with its own rootfs) yet: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=595926

hell9001 wrote at 2011-03-25 08:38:

"... was mit openvz in unstable passiert ist (was unglücklicherweise überhaupt nicht in den Changelogs des Kernel erwähnt ist, aber das ist eine andere Geschichte)."

Kannst Du kurz erläutern was Du damit meinst, oder einen entsprechenden Link posten.

thx hell9001

Gerfried Fuchs wrote at 2011-03-25 09:16:

In den Changelogs der Kernel-Paketen findet sich keinerlei Hinweis darauf, dass openvz entfernt wurde. Ich hatte nachgefragt und die Info bekommen, dass das aufgrund unterschiedlicher Entwicklungsbranches so sei und in dem Branch openvz nie drin war. Allerdings finden sich im Changelog durchaus Hinweise auf openvz ...

Comments are closed for this story.

If you want to syndicate this blog, feel free to do so.
This list contains the feeds that I follow:

Sun Mon Tue Wed Thu Fri Sat


©opyright 1999++ by Rhonda
[rss feed]

[html by vim] [graphics by gimp]

[generated by wml]

[powered by blosxom]