FreeBSD virtualization with ezjail

Motivation

I have some clients with old versions of python libraries and want to develop for them, but don't want to pollute my system and other pythons. So create isolated instances in FreeBSD jails, which seem similar to the OpenVz tool on Linux but not true virtualization (which I don't need here, just running FreeBSD).

ZFS

As an aside, on my FreeBSD-7.x system I'm using ZFS. I've got the usual /, /usr, /tmp, and /var mounted as "legacy" mounts, while /usr/local and everything else is a modern ZFS mount.  I like to put anything that's not strictly part of the operating system into /usr/local, so I'll want my jails there, instead of the default /usr/jails. Create the partition:

root@Boqueria:etc# zfs create tank/usr/local/jails

Ezjail

The ezjail installs from ports trivially.  It creates a default config file that says where jails should be created by default -- /usr/jails.  I prefer non-system stuff in /usr/local/ and I have a non-legacy ZFS filesystem there, so I change /usr/local/etc/ezjail.conf to specify /usr/local/jails.

ezjail_jaildir=/usr/local/jails

I referred to a number of write-ups I found on the net for ezjail; most were for earlier versions, not the 3.0 that was in my ports.

• http://www.scottro.net/qnd/qnd-ezjail.html
• http://erdgeist.org/arts/software/ezjail/
• http://www.joomlawebserver.com/freebsd/base-jail/11.-freebsd-jails-with-the-ezjail-tool.html
• http://wiki.freebsd.org/AppserverJailsHOWTO
• http://tomster.org/blog/archive/2005/11/10/using-ezjail
• http://erdgeist.org/arts/software/ezjail/

Before initializing my jail with a copy of the OS, I need to update my OS. I do this periodically from the STABLE tree. Currently I'm at FreeBSD-7.1-PRERELEASE which I built months back, so I'm probably due:

cd /usr/src
make -j4 buildworld
mergemaster -p
make -j4 buildkernel
make installkernel
[reboot, make sure it works]
mergemaster -is
make installworld
[reboot]

Yes, I should have a kernel file tuned for my AMD64 quad-core CPU, but I don't yet.

Since we've just built (and installed) a current world, we can use the "-i" flag to tell ezjail-admin it doesn't have to make world first, just install; we'll also tell it to install a ports tree (it's big, but will be shared amongs all jails):

ezjail-admin update -i -p

Add an IP alias for the new jail, in /etc/rc.conf:

ifconfig_em0_alias1="inet 192.168.255.40"

and set it on the command line

ifconfig em0 192.168.255.40 alias

Then create the new jail instance with the new IP. 

ezjail-admin create myjail 192.168.255.40

After copying a bunch of files, it complains about servers I'm running which listen on all addresses of my network interface, which could be a problem:

Warning: Some services already seem to be listening on all IP, (including myjail.mydomain.org)
  This may cause some confusion, here they are:
dovecot  imap-login 7376  4  tcp4   *:993                 *:*
www      httpd      1647  3  tcp46  *:80                  *:*
...many more...

It creates a /usr/local/etc/ezjail/JailInstanceName where you can see it assigns the IP address you provided upon creation.

Copy /etc/resolv.conf into your jail.

Add a startup config to the server's /etc/rc.conf:

ezjail_enable="YES"

then start up the jail system, then enter it:

$ sudo /usr/local/etc/rc.d/ezjail.sh start
$ sudo jail /usr/local/jails/myjail myjail.mydomain.org 192.168.255.40 /bin/sh
Password:
# pwd
/
# ls
.cshrc          bin             lib             proc            sys
.profile        boot            libexec         rescue          tmp
COPYRIGHT       dev             media           root            usr
basejail        etc             mnt             sbin            var
# svn
svn: not found
# emacs
emacs: not found

Well, it sure looks jailed. All my software development tools are locked safely outside. If I'm gonna be doing development, I have a lot of installation to do. Can I do this in a "flavour"?

Add a Flavour

For development I'm gonna need svn and emacs, perhaps a flavor can cause these to install. Maybe I can have the flavor do the critical /etc/resolv.conf too.

cp -Rp /usr/local/jails/flavours/default /usr/local/jails/flavours/swdev

I edited some stuff to .../swdev/ezjail.flavour script to pkg_add svn and emacs, remove the old jail, and created the new, but it hung for a while -- with no indication of what was happening.  I need to give it an /etc/resolv.conf.  Probably can't copy it from server, since the context of the shell script is the jail, so I'll just have to create by echoing text into it.

echo -n '$1$p75bbfK.$Kz3dwkoVlgZrfLZdAXQt91' |\
 pw useradd -n chris -u 1001 -s /bin/sh -m -d /home/chris -G coders -c 'Chris Shenton' -H 0

echo "domain    mydomain.org"            >  /etc/resolv.conf
echo "nameserver        192.168.255.5"  >> /etc/resolv.conf

cp /usr/share/zoneinfo/America/New_York /etc/localtime

pkg_add -r emacs
pkg_add -r subversion
pkg_add -r bash
pkg_add -r sudo
chpass -s /usr/local/bin/bash chris

Then we remove the old, and add it again, this time specifying the flavour:

ezjail-admin stop myjail
ezjail-admin delete -w myjail
ezjail-admin create -f swdev myjail 192.168.255.40
ezjail-admin start myjail

It does take some time now to startup the jail the first time as it has to fetch and install those packages -- about 5 minutes on my system for those four packages. The next time the jail is started, it comes up instantly. 

It would have been easier in my flavour to create an /etc/resolv.conf file with my nameserver info.

You probably want to enable ssh in the flavour's /etc/rc.conf file.

Let Jails Accept Connections

Ezjail warned us of daemons that were rudely/naively listening on all interface addresses. If we want to be able to -- say -- ssh into a jail, we have tell the main server to be not so promiscuous.  For ssh, I did this in /etc/ssh/sshd_config, then restarted it:

#ListenAddress 0.0.0.0
ListenAddress 192.168.255.100

I'll have to do likewise for apache and others.

Edit the jail's /etc/rc.conf to enable sshd. Then enter the jail and start it up.

sudo jail /usr/local/jails/myjail myjail.shenton.org 192.168.255.40 /usr/local/bin/bash
[root@myjail /]# /etc/rc.d/sshd start
Generating public/private rsa1 key pair.
...

Now you can ssh to it with its own jailed IP address (and name if you've entered it into DNS).

Uh, you really should give root a password. :-)

Adding More Jails

I created a new flavour called 'minimal' for minimal installs, where I will manually add my own packages. (My first use will be for zenoss, which needs python and mysql and some others).

ezjail-admin create -f minimal zenoss.mydomain.org 192.168.255.41
ezjail-admin start zenoss.mydomain.org