[SGVLUG] Setting up lightweight containers of various flavors of Linux?

[Dan Kegel]

I'd like to be able to run build and test scripts using lightweight
containers of some sort, where the host is ubuntu 12.04, and
the guest is RHEL 5/6 and ubuntu 10.04/12.04 (and probably more later).

Here's a little brain dump from about two hours of exploration.
You can stop reading now if you don't like line noise.

There are various flavors of tools that let you fake chroots,
like fakeroot, fakechroot, fakeroot-ng, and scratchbox's network fakeroot;

various flavors of tools to manage real chroots,
like schroot and dchroot;

and various flavors of tools to do something stronger than chroots,
like lxc-tools, libvirt, and systemd-nspawn (all driving lxc), and of
course kvm.
Use kvm if you need the guest to run a different kernel than the host,
or if you need security;
http://berrange.com/posts/2011/09/27/getting-started-with-lxc-using-libvirt/
says "repeat after me: lxc is not secure.  If I want real security I
will use KVM." ).

Orthogonal to those, there are various flavors of tools to
set up working systems inside those containers,
like debootstrap, cdebootstrap, febootstrap, rpmstrap, rinse,
xen-tools, mock, and mach.  And virt-install.

All of which are rumored to be out of date :-)

So, what's the state of the art, and can actually handle creating and
managing lxc guests containing RHEL 5/6 and Ubuntu 10.04/12.04?

It's tempting to go with libvirt simply because then if I had to use
actual virtual machines, I wouldn't have to change much.

virt-install's support for installing in lxc seems recent, see
https://bugzilla.redhat.com/show_bug.cgi?id=693876  and
http://lists.fedoraproject.org/pipermail/virt-maint/2011-July/002636.html
http://berrange.com/posts/2011/09/27/getting-started-with-lxc-using-libvirt/

I'm less than thrilled about having to use kickstart or preseeding
instead of debootstrap, but maybe there's no other way of reliably
installing recent rpm distributions?

There are scripts like lxc-ubuntu, lxc-debian, and lxc-fedora to create
various kinds of systems.  These may have been replaced by
lxc-create, which has a -t option to indicate guest os type,
but some doc I've seen for lxc-create is scary.

lxc-start boots up services inside an lxc container; lxc-execute is lower
overhead, and just starts one app.  virsh probably needs lxc-start to
have been run.
Don't know how long startup takes.

It looks like older ubuntu versions can't run in lxc without some tweaking:
http://www.stgraber.org/2012/05/04/lxc-in-ubuntu-12-04-lts/
http://www.stgraber.org/2012/03/04/booting-an-ubuntu-12-04-virtual-machine-in-an-lxc-container/

febootstrap seems quite cool, but is aimed at systems where the
host and the guest are running the same OS.

cdebootstrap seems to be debootstrap reimplemented in C,
but it's less well maintained, and probably can be ignored.

rinse is like debootstrap for rpm-based distros.  It handles up to date centos,
but only rhel-5 (I sense a patch opportunity).
On the other hand, just using yum seems to be an option, too:
http://geek.co.il/wp/2010/03/14/how-to-build-a-chroot-jail-environment-for-centos
so it's not doing that much magic.

mock crashes on startup on my ubuntu 10.04 system and appears to be
AWOL on ubuntu 12.04.

dchroot is said to be obsolete.  schroot seems current.

Supposedly lxc containers are fragile with respect to apt-get update
inside the guest?
http://johnlewis.ie/using-schroot-instead-of-lxc-containers/

chroots are dangerous, if you have symlinks into the host, you can
accidentally delete
your whole system when trying to remove the guest (I've done it!).

So, it seems like things haven't changed too much since I wrote that
little helper script http://kegel.com/linux/web32.sh.txt:
  - use schroot to manage a real chroot jail
  - use debootstrap to initialize Ubuntu images, and rinse to
initialize RHEL images
  - then use either lxc-execute or schroot to run an isolated command
in an unstarted container, depending on how isolated you want it

I guess I'll start with schroot until I find a reason to use lxc.