First time installing TinyDNS as a service in a FreeBSD jail - how to do it?

Question

I'm using FreeBSD 11.2 at the moment (likely to move to 12 in a while). I need a tiny authoritative-only DNS server (no lookup or caching needed, under 10 domains and under 10 queries/hr, almost no record changes).

I'm likely to go with TinyDNS, part of djbdns, which is well reputed for security and also seems minimal and does what I need.

Part of the reason for security is that it'll be exposed to the internet, although behind an IP/port filter and very low rate, rate limiter (using pf for this). But for that reason, I do want to take especial care on how the daemon is set up, to avoid obvious vulnerabilities. By that, I mean aspects such as the needed users and groups, start/stop scripts, jail/chrooting, and minimising/disabling/denying the main non-essential access/capabilities that an attacker could pivot.

(I should mention that I can install tinydns "ordinarily" on a test system, and create the needed .conf file, so it's purely how to run it in a secure manner, that's missing)

I'm not experienced in setting up software to run chrooted/jailed, or reviewing a chrooted/jailed package for appropriate security practices, and this'll also be my first time trying to do something like this, although I've chosen the specific DNS server package specifically for its apparent setup simplicity.

Ignoring the .conf file, what would a "recipe" look like, to set TinyDNS up to run correctly as a service, and ideally with minimised access to "other stuff not essential for the daemon but helpful to an attacker"?

Answer 1

This is rather lenghty so the very short version for those who cannot be bothered to read it all:

echo "testjail { }" >> /etc/jail.conf
mkdir -p /usr/local/jails/testjail
bsdinstall jail /usr/local/jails/testjail
service jail start testjail

pkg -j testjail install -y nginx
sysrc -j testjail "nginx_enable"=YES
service -j testjail nginx start

I would typically rate this question as "too broad" and refer people directly to the FreeBSD Handbook. But I myself find that section to be rather poorly written and confusing. It is all there - but things are much easier than they seem! I just wish they focused more on the concepts rather than listing commands. And you might have fun reading Jails – High value but shitty Virtualization

What I will do instead is to humbly outline what I personally do and what stumbling blocks I have met. My fails might not be yours, but as I once was in the same situation as you, I hope my journey can help. The better you understand FreeBSD overall the easier jails become.

What is a jail?

Many descriptions gloss over the important part of what a jail is. It helps immensely to understand what the kernel does. It "runs" code and keep track of PID (Process ID) and UID (User ID). This is then common knowledge for many Unix users. The FreeBSD kernel has then added the concept of a JID (Jail ID). The kernel is then able to partition processes into jails. This in effect means that the FreeBSD kernel is able to "virtualize" systems without any overhead. You still only have one kernel but can have multiple systems. This was key for my understanding of the concept.

If you run a vanilla box "without" jails. Then all processes belong to JID 0. When you start playing with jails we then call this the "jail host".

With that in mind the next step for me was to understand the connection with how FreeBSD actually works. If you come from a Linux background you probably know that Linux is "only" the kernel. What makes the system is then the userland supplied with the distribution (Ubuntu, Debian, Slackware etc.). FreeBSD is the combination of the kernel and userland. It is the full OS (Operating System).

So a very crude summary of Chapter 12. The FreeBSD Booting Process is:

1. BIOS/UEFI Bootstrap
2. FreeBSD Bootloader
3. FreeBSD Kernel
4. FreeBSD init runs rc scripts and all that magic

You might know how modular FreeBSD is and that you could replace the rc system with something like OpenRC but we ignore that for now to keep it simple.

When you then start (boot) a jail the kernel then does init with a new JID (ie. 1). Everything here is then confined to jail 1. What init needs to touch must be within the chroot'ed filesystem. If you want to keep things simple that is then the full userland. But we want to keep the jail seperate from the host system so this is then a full copy of the userland!

The importance of how these things are interconnected cannot be understated. When you fully grok it you will notice that jails are directly supported by many FreeBSD tools. Not just simple things as ps -J but bsdinstall, freebsd-update and pkg! If maintaining a FreeBSD system is second nature to you then jails will be a walk in the park!

But for most of us we are somewhere inbetween on that path to Nirvana and might have to struggle with some of the concepts to get it done "right".

hier

I am a huge fan of hier. It gives a clear an consistent view over where things should be placed. Unfortunately the powers that be never made any decision on what a nice default location for jails would be. When you combine this with a lot of different tutorials that use different locations and mix in various jargon as "base install" and "templates" things get confusing fast!

The Handbook does not discuss this and just refer to /here/is/the/jail. And in the final example they use /home/j and /home/j/mroot. I prefer to keep user directories and only user directories in /home. And simply using a shorthand notation like j is a big no-no in my book.

I would say that the most common and "correct" location used is /usr/local/jails. A strong contender mostly for people using ZFS is /jails

In this location I would put the chroots for my jails. Thus I have spoken. Hencefort jails shall be found here.

I found this very confusing when I started out. Giving me freedom to place it anywhere made me more insecure as I did not understand the consequences.

To add to this confusion many tutorials was using "basejail", "skeleton directories", "templates". This added to the confusion of what a jail really is and conflated it with a lot of management.

Filesystem

We do not really care what filesystem we are using. It can be UFS or ZFS. For the sake of the jail we only need a directory for our chroot.

When using UFS the important thing to note is how you have arranged your slices. The beginner has often not given this much thought. Which slice have enough space to accomodate the chroots. This is why I specifically do not like to use /home for this purpose. Maybe you create a slice for the purpose or you just create a directory somewhere and then later figure out if you have space enough.

If you are using ZFS the issues are actually the same. Because of the way ZFS is structured you could argue that it is better to create a new dataset rather than just doing mkdir. But for the beginner you should not worry. Doing a ´mkdir´ with ZFS is better for you. This makes things simpler and you can move on without ZFS.

The next problem is that many tutorials also operate with a "basejail" which is a full vanilla system ready to copy to create a new jail. Some tutorials will then tell you to do a ZFS clone rather than doing a copy. A long the way you will update the basejail and naturally make snapshots of it. But then you notice that you cannot delete any snapshots which have clones of them. So in actual use I prefer to use zfs send/receive.

In my mind tutorials should only touch on ZFS as an addendum. ZFS is a huge topic on itself and should be treated like that. When you are comfortable with both jails and ZFS then you can combine to your hearts content and reap the benefits. If not you end up building a very brittle tower without any ability to fix things when they break.

fat/thick vs thin

The FreeBSD Handbook talks about “complete” jails and “service” jails. Most other places uses the terms thick (fat/full) and thin jails. The thick and thin jails are both "full" virtualized FreeBSD systems which The FreeBSD Handbook refers to as "complete" jails. The "service" jail is however much more elusive

A thick jail is then a copy of the full operating system. But how much fat is that? FreeBSD 11.2 with base/lib32/ports weigh in at 1.4G but for a jail, "base" alone would likely do which keeps you at a relatively svelte 512M (Compared to the 20G of C:\Windows for Windows 10).

A thin jail is then a bag of trick to minimize the on-disk usage of a "complete" jail. By using nullfs (a loopback file system sub-tree) to mount a read-only filesystem and a couple of well placed symlinks to enable read/write parts you slim down the disk usage. Unfortunately I do not have any numbers at hand. When you have the on-disk structure in place then you simply add mount.fstab to your jail.conf to mount the filesystem when starting the jail.

I got the ideas how to do this from FreeBSD Jails the hard way. This taught me how to handle jails without any 3rd party utilities and finally made the pieces click for me. And despite the title - it is actually the easy way. They did however get one thing wrong. You should not do a zfs clone but rather a zfs send/recieve at described in Unadulterated Jails, the Simple Way. Both have probably found inspiration in the slightly outdated Multiple FreeBSD Jails with nullfs. Another worthwhile source is FreeBSD Thin Jails. Finally a newer source which also covers VNET (we will get back to that!) is How to configure a FreeBSD Jail with vnet and ZFS

The above is commonly referred to as a thin jail but it could be done in other ways. All roads lead to Rome and you decide for yourself how you want to implement it locally.

All this leads us to the elusive "service" jail. A jail which just runs one specific service in the most locked down scenario. Just the actual application and only a thin layer to support it. It can be done but I have not seen a lot of work on it.

The typical jail.conf would contain:

exec.start = "/bin/sh /etc/rc";            # Start command
exec.stop = "/bin/sh /etc/rc.shutdown";    # Stop command

This is what ensures that the rc system is run a jail when it start/stop. If you have an extremely simple executable you could simply point to that instead. So the question is: How much of the OS does my service need access to? What shared libraries are needed and do you run scripts?

I know of no work which has done this for you. So you start with a full jail and then removes what is not needed for your particular service. Many command-line utilities such as top, ps and tail are safe to remove as they are typically not used by a daemon. But you might miss them if you are debugging within the jail yourself. If you start a daemon directly with exec.start you do not need rc.subr and friends.

If you go that route and start a killing spree to determine how much can be removed from the OS (to reduce attack surface) while the service remains functional then you should be aware of nanobsd. This can help you tailor the installation to your needs.

So - definitely doable. But I know of no public work. Most people go with thick or thin jails and just run their designated service there.

Dependency hell

When doing thin jails you need to be very careful to do everything correct or things will break. When you update the system you need to remember to update all the different parts. The basejail or sources you started from. The templates and the actual jails. This is a lot of moving wheels which makes management harder. It can be done but you should weigh the effort against the prize. Disk space is very cheap - so you you would need quite a few services to make it worth the effort.

I would then recommend by having simple fat jails with a full OS within.

If you are very comfortable with ZFS then by all means go wild. But if not then I would strongly suggest to have the chroot in a simple directory. When you feel comfortable working with a jail then you can start adding the cream.

The key is then to treat the jail as a seperate system. If you maintain your system properly today you should already have the habit of using freebsd-update fetch install and you will know the it updates the kernel and the userland.

When adding jails to the mix you just need to remember to update those as well:

freebsd-update -b /usr/local/jails/testjail install

If you are doing upgrades - then remember the jails as well:

 freebsd-update -b /usr/local/jails/testjail --currently-running 10.3-RELEASE -r 11.0-RELEASE upgrade

Just the same way as you keep your packages fresh

 pkg upgrade
 pkg -j testjail upgrade

Hey, man! I just wanted to jail a service!

OK! With all the caveat emptors let's get this done the easy way on a totally vanilla system. TinyDNS is a bad example so let us just install nginx

1. Create a zane jail config in /etc/jail.conf
2. Allow jails to start on boot
3. Add the jail name "testjail"
4. Add the directory for the "testjail" chroot
5. Install the OS into the chroot (Choose only base. De-select lib32/ports)
6. Start the jail
7. Add nginx package to the jail.
8. Tell the jail to start nginx at startup
9. Start nginx now!

Done!

1 & 2 only for getting ready for jails. 3 - 6 for each new jail. 7 - 9 for each service.

cat <<'EOF'>/etc/jail.conf
# Global settings applied to all jails.
host.hostname = "${name}.jail";
ip4 = inherit;
ip6 = inherit;
path = "/usr/local/jails/${name}";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

# Specific seetings can be applied for each jail:
'EOF'
sysrc "jail_enable"=YES

echo "testjail { }" >> /etc/jail.conf
mkdir -p /usr/local/jails/testjail
bsdinstall jail /usr/local/jails/testjail
service jail start testjail

pkg -j testjail install -y nginx
sysrc -j testjail "nginx_enable"=YES
service -j testjail nginx start

This is what I would claim to be the "correct" way of doing a jail and simple management. There are many variations - the last line could be replace with service jail restart testjail. At this point I am able to browse web page on my jailed nginx instance.

Some FreeBSD defaults do not lend itself so well to jails. So consider setting these as well:

sysrc -j testjail sendmail_enable="NONE"
sysrc -j testjail sendmail_submit_enable="NO"
sysrc -j testjail sendmail_outbound_enable="NO"
sysrc -j testjail sendmail_msp_queue_enable="NO"

Remove a jail

To delete a jail:

1. Stop the jail
2. Remove the jail name from ´/etc/jail.conf`
3. Reset flags so you are allowed to delete the files
4. Delete the files

service jail stop testjail
sed -i '' '/^testjail {/ d' /etc/jail.conf
chflags -R noschg /usr/local/jails/testjail
rm -rf /usr/local/jails/testjail

The above is to show that it is not hard to manage jails with the tools at hand and without the need for at 3rd part jail management tool.

Hey, man! I just wanted to jail TinyDNS!

You write you could install tinydns "ordinarily" - hence I will leave that to you. For others who might be following they would probably guess that it should be as simple as:

pkg -j testjail install djbdns

But whoever created that package was not nice enough to create a service wrapper for tinydns in /usr/local/etc/rc.d. You then need to figure out how to run it as a daemon. You could choose to use alternate service management such as supervisor or a simple hack as adding it to /etc/rc.local. Or for brownie points create a rc script and contribute it back for all of us to enjoy!

This leads us to the only two specific jail commands you really need to know: jls which lists running jails and jexec which allows you to execute stuff inside the jail.

root@test:~ # jls
   JID  IP Address      Hostname                      Path
     3                  testjail.jail                 /usr/local/jails/testjail

To get a command-line as root we simply execute the root shell (remember that is tcsh!)

root@test:~ # jexec testjail /bin/tcsh
root@testjail:/ # tinydns
tinydns: fatal: $IP not set
root@testjail:/ # exit
exit
root@test:~ #

When working within that shell everything is within your jail/chroot.

What they do not tell you!

Up to now I have not strayed much afar from what most tutorials will tell you. After wasting a lot of time in the filesystem they skip the exciting part. Today there is not much fun with computers without network access. In the above example we shared the network stack with the host. This means that you cannot run anything on the jail host which utilizes the same ports as a jail. In my example this was port 80 for the webserver.

On the other hand it is now very easy to setup a firewall. You simply treat all ports as local. If you are using IPv6 things are a breeze. But most (all?) of us still need to struggle with IPv4. IPv4 is simple enough but you probably do not have the addresses your need and then we need to resort to some sort of NAT.

If you want to get really fancy with networking we have virtual network interfaces VNET. It has matured over the years but to use it you need to compile a kernel with VIMAGE support. This is enabled in the generic kernel from FreeBSD 12.0 and onwards.

When using VNET you have a nice virtual interface where you can run your firewall inside the jail.

I do currently not use VNET as a unified firewall on the jail host is enough for me and what I want. With this I control which traffic goes to and from the jail. This makes it even harder break things from the jail.

On the jail host I often allow some outbound traffic (http/ftp/nameresolution). But all traffic in my jails I specifý exactly which ports are allowed in both directions and between jails.

The trick is to move the traffic to another interface. The local loopback interface lo0 is a great candidate. But to keep the rules better seperated it is even better to clone it to a new interface name lo1. You can set this up and prepopulate the IP addresses for this in the jail host using ifconfig. There is however no need as the jail subsystem handles all this for you automatically.

To do this our /etc/jail.conf now looks like this:

# Global settings applied to all jails.
host.hostname = "${name}.jail";
interface = "lo1";
path = "/usr/local/jails/${name}";
mount.fstab = "/usr/local/jails/${name}.fstab";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

# Specific seetings can be applied for each jail:

testjail { ip4.addr = 172.17.2.1; }

Notice how ip4/6 = inherit; has changed to interface="lo1". Notice also mount.fstab - this is how I add my nullfs mounts but I will not go further into this now. Then I have added the internal IP I want to use to testjail.

We then clone the by adding lo1 to /etc/rc.conf:

sysrc "cloned_interfaces"="lo1"

You need to reboot for this. If you want to try it out now without a reboot you need to create the interface yourself:

ifconfig lo1 create
ifconfig lo1 up

This has the same effect as setting cloned_interfaces="lo1" in /etc/rc.conf. You do not need to create aliases for the IP addresses as this is handled automaticly when the jail starts.

This is however boring as traffic gets nowhere.

To get the traffic going we need to set up the firewall and get some NAT going. My poison of choice is pf. You have your ruleset in /etc/pf.conf

To NAT from the outside to a jail IP for http do like this:

rdr pass inet proto tcp from any to (em0) port http -> 172.17.2.1 port http

Many services do not like to start if they are not able to connect to themselves. Hence I add a rule for that as well.

pass on lo1 proto tcp from 172.17.2.1 to 172.17.2.1 port http

The NAT rule is only needed if you want someone from the outside to connect. When you start to have multiple jails you often just want one jail to be able to connect to another.

pass on lo1 proto tcp from 172.17.2.1 to 172.17.2.2 port 8180

With these simple firewall setting you can have a very nice network isolation between your jails and the outside.

Summary

From what I have experienced myself I would suggest the following path to understand how to use jails:

1. Understand basic kernel functionality - what is a process
2. Understand chroot
3. Understand the boot process
4. Build a fat jail
5. Work with the jail. Use/update
6. Understand hier
7. Build a thin jail
8. Work with multiple jails
9. Understand networking
10. Setup Networking
11. Understand ZFS

That is my recipe. Probably not quite what you hoped for :-)

If you do not want to handle the gritty bits with the supplied tools then have a look at some of the 3rd party tools:

Bastille

ezjail

iocage

From all this you can see that your question was rather broad and depends heavily on your preferred setup. I believe I have shown a receipe for a well behaved package such as nginx but maybe you need to ask some other questions:

• How do I daemonize tinydns?
• How do I run tinydns as an unpriviledged user?
• How do I write a rc script

But then I would leave the jail parts out of it.