in late may, we (again) visited val bregaglia for another very nice vacation. the weather was great, except on monday when it snowed. we stayed once more in vicosoprano. we did some hiking from soglio to castasegna, and from castasegna to chiavenna:
in april, we visited the beatushöhlen near interlaken, a stalactite cave where you can walk through.
the aim of this post is to describe how to set up an encrypted arch linux installation on a headless server. while migrating to a new server during the last days, i had to go through the procedure another time. since it is easy to screw something up and you don't get helpful error messages without a serial console or (virtual) kvm, i wanted to share my instructions on how to set up such a machine. my previous server, hosted at strato, had a serial console via ssh included, so it wasn't that challenging to set it up. for my new server, hosted at hosttech, no serial console is available, but you can get a kvm attached. i had my kvm day yesterday as it makes life much easier (handling grub menus, or see what went wrong when networking doesn't work), and set up the machine twice to see whether i could also do it without a kvm. the instructions here now work without a serial console or kvm, though ymmv: tiny differences in systems, rescue boots etc. can send you into a situation where something doesn't work and you don't know what. so be warned, and try it out with a vm first to be on the safe side. doing this whole thing with another distribution is certainly also possible, but will in many details be substantially different from what i describe here. these instructions also contain some hardening not necessarily for all situations.
this post assumes you have a certain level of linux experience. i assume that you have a headless server sitting somewhere which has a software raid-1 disk configuration and you have a rescue system available which boots over the network. all dedicated server hosters i know provide something like that, you can usually set a flag in the customer/setup area of your hoster to start such a system on the next boot. hosttech uses riplinux for their rescue system, so some of the details i describe below might be specific to this one and not work with other such systems.
your server will end up in a state where you have to unlock the encrypted disk remotely via ssh, so as long as your server isn't compromised (which can happen if it is hosted at a place you don't control), you can unlock it after reboots without entering your password in a kvm/serial console (which might be tapped into). this also means you must unlock it after every reboot; it won't come back up alone by itself. (otherwise the encryption would be moot.) so don't put anything on the server which is too critical to be leaked. (you might not want to put it on a computer in the first place, though.) despite this disadvantage, one big advantage is protection of your data: if a faulty disk of your server is replaced, or your server is decommissioned, your data cannot be extracted from the disk without knowing your encryption key. and if you can wipe the luks header several times, even having your keys won't bring the data back (except if you have a backup of the header and the person having access to your key also has access to that backup).
- the whole setup is split up into two parts:
- setting up a small unencrypted installation of arch linux on the server;
- using that unencrypted installation to set up a proper encrypted arch linux server.
i've chosen this approach for the strato server back then since strato's rescue system didn't offer
cryptsetup/luks back then. this approach also has less requirements on the rescue system, and you have a clean arch linux install to set up the real system. and you can use the unencrypted installation as your own personal rescue system to do maintenance on the encrypted installation, and be sure that all necessary tools are either already installed, or can easily be added the same way as you usually install packages on your real server. (rescue systems don't have to offer a package manager, so installing something you need but which isn't there can be really annoying.)
one simple note before we begin: if you need to create a password or random text string, you can use use
dd status=none if=/dev/random of=/dev/stdout bs=1 count=15 | base64 to generate them.
also note that the arch linux wiki has a collection of useful installation guides, which cover a lot of different cases. here, i'm mostly following the steps in install from existing linux, as well as instructions from remote unlocking of the root (or other) partition. the wiki also contains a huge amount of other useful information, like howtos on setting up encrypted systems in many different variants.
one final note: you might be tempted to also try to encrypt the boot partition; while this is possible nowadays, you cannot use it for your server, as for remote unlocking you need the init ramdisk up and running, whose contents are stored on the boot partition. this will change if at some point, grub will include a possibility for remote unlocking. (if that ever happens.) (what you could also do is create a mini boot partition which allows remote unlocking the real boot partition, and then boots the system installed on the real boot partition. that doesn't really improve security by much, though.)
as all such instructions, this post comes without any warranty. you're on your own! if you have data on the server, back it up first! these instructions will delete everything on your server, and might put it into a state where it must be reset by your hoster, which might cost you money. also, if your server is currently a production machine, be sure that it is no longer actively used and all data is backed up before you start playing. if something goes wrong, don't blame me.
setting up the unencrypted arch linux installation
- first boot your server into the rescue system, and begin setting up partitions. you need (at least) three partitions:
- a partition for the unencrypted install (2 gb);
- a boot partition for your encrypted install (2 gb);
- a partition for the encrypted partitions (rest).
in case you use a gpt partition table, you need a bios boot partition. if you're using uefi, you'll have to ask someone else (and probably adjust some more things in my instructions, so try it out in a vm or with a serial console/kvm first!).
next, create raid arrays for partitions 2—4 (i'm assuming 1 is a bios boot partition; if not, you have to renumber the devices below):
mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sda2 /dev/sdb2 mdadm --create --verbose /dev/md1 --level=mirror --raid-devices=2 /dev/sda3 /dev/sdb3 mdadm --create --verbose /dev/md2 --level=mirror --raid-devices=2 /dev/sda4 /dev/sdb4
the next step is to create an ext4 filesystem on
/dev/md0 which will serve as the root filesystem of the unencrypted system:
mkfs.ext4 /dev/md0 mount /dev/md0 /mnt
/dev/md1 will later host the boot partition of the encrypted system, and
/dev/md2 will store the encrypted root, home and swap partitions (or whatever more you want to create). it is good practice to wipe the encrypted partition, either before creating the encrypted system (by filling it with random data) or afterwards (by filling the encrypted partition with zeros). to wipe the partition before encrypting it, you can run:
openssl enc -aes-256-ctr -pass \ pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" \ -nosalt < /dev/zero > /dev/md2
note that the hosttech riplinux rescue system has no base64; you can instead run
dd if=/dev/random bs=128 count=1 2>/dev/null | base64 on your desktop computer and put the result into the double quotes above. this step is rather slow, so you'll better do it in a screen session. on my server, it took roughly 1.5 hours for a 500 gb partition. in fact, starting a screen session is a good idea anyway, as you don't want connection failures to interrupt (and potentially destroy) your installation procedure.
to set up arch linux on
/dev/md0, i followed the instructions here with some modifications; most of them were because the rescue system didn't support certain features. here are the details of what i did:
cd /tmp wget http://mirrors.kernel.org/archlinux/iso/2016.06.01/archlinux-bootstrap-2016.06.01-x86_64.tar.gz sha512sum /tmp/archlinux-bootstrap-2016.06.01-x86_64.tar.gz
(the original instructions use
curl -O instead of
wget, but riplinux only provides the latter. also, the original url is
https://, but the provided
wget couldn't connect.)
i also downloaded
on my desktop machine, computed
archlinux-bootstrap-2016.06.01-x86_64.tar.gz and compared it to the one on the server, and finally used
gpg (gnu privacy guard) to verify the signature (see this document for details on signature verification). if the sha512 checksums match and the signature validates, everything's ready to go! (you might have to use
sha256sum or even
md5sum, depending on what the rescue system you're using offers. if your rescue system offers
gpg, you can also validate the signature on the server itself without downloading the file a second time.)
next, continue with:
tar xzf archlinux-bootstrap-2016.06.01-x86_64.tar.gz
now you're supposed to run
/tmp/root.x86_64/bin/arch-chroot /tmp/root.x86_64/ according to the instructions, but that didn't work on any of the rescue systems i tried. instead, the manual method works:
mount --bind /tmp/root.x86_64 /tmp/root.x86_64 cd /tmp/root.x86_64 cp /etc/resolv.conf etc mount -t proc /proc proc mount --rbind /sys sys mount --rbind /dev dev #mount --rbind /run run chroot /tmp/root.x86_64 /bin/bash
i skipped mounting
/run as it wasn't provided on the rescue system. everything works fine without it. the next step is to set up
pacman, the arch linux package manager. the suggested step for this is
pacman-key --init which generates a gpg key using random data from
/dev/random. unfortunately, on a headless server, this takes a long time. if you can, you can speed this up using haveged if your rescue system provides it, or you generate the necessary files on another system. to do this on my local machine, i downloaded the above bootstrap archive (
archlinux-bootstrap-2016.06.01-x86_64.tar.gz), extracted it, chrooted into it, and ran
pacman-key --init there. (it was done after a few seconds, as opposed to the 8 hours i tried on the headless server first, after which i killed it.) go into
root.x86_64/etc/pacman.d and do
tar cf pacman.tar gnupg, and transfer
pacman.tar onto the rescue system. on the rescue system, go to
/tmp/root.x86_64/etc/pacman.d/ (outside the
chroot, as the
chroot provides no
tar!) and extract the tarball there, so that you now have a non-empty subdirectory called
then go back into the
chroot and continue with:
pacman-key --populate archlinux
next, leave the
chroot and edit the mirrorlist at
chroot provides neither
nano, but the rescue system does). uncomment whatever mirror you find useful and go back into the
chroot environment. make sure that some
http:// mirrors are uncommented as well if you had problems with downloading the
https:// bootstrap archive above. then set up the basic system inside the
pacman -Syy pacman -S base base-devel parted
the next steps in the official howto is to continue with
pacstrap and later
arch-chroot. that didn't work for me; both scripts complain about
devtmpfs not being available, and
arch-chroot also complained about the invalid argument
unshare. i patched the scripts with
nano `which pacstrap` nano `which arch-chroot`
by searching for
devtmpfs twice (the first ocurrence is at the beginning); the second match should be at these two lines:
chroot_add_mount udev "$1/dev" -t devtmpfs -o mode=0755,nosuid && chroot_add_mount devpts "$1/dev/pts" -t devpts -o mode=0620,gid=5,nosuid,noexec &&
i changed these to:
chroot_add_mount -o bind /dev "$1/dev" && chroot_add_mount -o bind /dev/pts "$1/dev/pts" &&
note that this will screw up the unmount mechanism in these scripts. that isn't nice, but it'll work without. (and as soon as you have the unencrypted system set up, you can use it to install the encrypted system, and since the unencrypted system is a full arch linux system, you won't have such problems again. that's another reason why i like to set up an unencrypted system as well.) in
arch-chroot, i also had to change
SHELL=/bin/sh unshare --fork --pid chroot "$chrootdir" "$@"
SHELL=/bin/sh unshare --fork chroot "$chrootdir" "$@"
i.e. remove the
--pid argument. finally, do
in case your rescue system doesn't have
/run/shm (like mine did). then you can proceed with installing arch linux. first, mount the partition you want to install the unencrypted system on as
mount /dev/md0 /mnt
then you can set up the base system:
pacstrap /mnt base genfstab -U -p /mnt >> /mnt/etc/fstab
note that on my system, this didn't use uuids for identifying the disks, which is in general a good idea. to find out the uuids for the devices, run
blkid and change
/mnt/etc/fstab by replacing entries such as
after that, continue with:
arch-chroot /mnt echo unencrypted-rescue-system > /etc/hostname ln -s /usr/share/zoneinfo/Europe/Zurich /etc/localtime echo en_US.UTF-8 UTF-8 > /etc/locale.gen locale-gen echo LANG=en_US.UTF-8 > /etc/locale.conf
obviously, you should replace
Europe/Zurich and possibly also
en_US with something more fitting. next, run
to set a root password. generate a random one and write it down in a safe place. (you can also later log in with
ssh and change it, if you fear the rescue system is too nosy.)
next, you have to configure your networking. first, you have to find your systemd network device name. they are usually of the form
enpXsY (assuming you don't use wlan for your server); to find the right name (your rescue system might use old
ethX names), run
lspci and look for
Ethernet controller. if you find something like
XX:YY.x Ethernet controller
you can extract
enpXXsYY right away. two caveats though: first, you need to strip leading zeros, and second, the numbers given by
lspci are in hexadecimal notation, while the ones in
enpXsY must be in decimal, so you'll have to convert them.
as soon as you found out the name of your network interface, create
/etc/netctl/wired with the following content:
Description='main ethernet connection' Interface=enpXsY # REPLACE THIS! Connection=ethernet IP=static Address=('xxx.xxx.xxx.xxx/yy') Gateway='xxx.xxx.xxx.xxx' DNS=('xxx.xxx.xxx.xxx' 'xxx.xxx.xxx.xxx') IP6=static Address6=('2001:xxxx:xxxx:xxxx::1/64') Gateway6=('fe80::1')
you need to adjust the interface name, the ipv4 and ipv6 addresses and the network masks and dns servers correctly, obviously. you can also use dhcp if your hoster supports that. next, continue with:
netctl enable wired pacman -S openssh grub lvm2 systemctl enable sshd.service
then you have to edit
/etc/mkinitcpio.conf and insert
mdadm_udev in the
HOOKS = "..." line somewhere before
filesystems. (otherwise, the system won't come up again as it won't be able to assemble the raid arrays.) next, edit
/etc/ssh/sshd_config and add
at its end. (otherwise you won't be able to login to the system at all, as
root is the only user.)
mdadm -E --scan >> /etc/mdadm.conf mkinitcpio -p linux grub-install --target=i386-pc /dev/sda grub-mkconfig -o /boot/grub/grub.cfg sync
finally, unmount and reboot:
exit cd / umount -R /mnt exit reboot
the unencrypted "rescue" system should be ready to go.
setting up the encrypted arch linux installation
log into the newly set up unencrypted system via
ssh root@your-server and your root password you set above. (now is the time to change it if you don't trust the rescue system too much.)
now contine by installing two important packages and creating a temporary filesystem:
pacman -yS cryptsetup screen mount -t ramfs -o size=1M none /mnt
screen session and continue in there. we'll need the temporary filesystem to transfer the master key for the encrypted partition without writing it to disk. instead of creating the master key on the headless server (which doesn't have enough entropy, probably), create it in your desktop computer:
dd if=/dev/random of=server-masterkey bs=1024 count=1 scp server-masterkey root@your-server:/mnt
back on your server, inside the screen session, create the encrypted disk:
cryptsetup --verbose --cipher aes-xts-plain64 --key-size 512 -h sha256 -i=10000 \ --verify-passphrase --master-key-file /mnt/server-masterkey luksFormat /dev/md2
you have to enter a passphrase for your encrypted partition. use a longer randomly generated password and store it safely, or something which is long enough and you can remember. the setting
-i=10000 i used is rather paranoid: hashing the password takes roughly 10 seconds on your server. this makes most brute-force attacks impossible, but also makes unlocking (and all other
cryptsetup operations) slow. feel free to decrease the number, but since these operations need to be done only very seldom (like now while installing, and once after each reboot of your server) there's no need to do that.
then get rid of the master key temp filesystem and open the encrypted partition:
umount /mnt cryptsetup luksOpen /dev/md2 cryptdisk
if you didn't wipe the space occupied by the encrypted partition earlier with random data, you can now run
dd if=/dev/zero of=/dev/mapper/cryptdisk bs=1M. (do that inside a screen session, as it will take a lot of time!)
create a lvm on the encrypted partition:
pvcreate /dev/mapper/cryptdisk pvdisplay vgcreate server /dev/mapper/cryptdisk vgdisplay lvcreate --size 32G --name root server lvcreate --contiguous y --size 4G --name swap server lvcreate --extents +100%FREE --name home server lvdisplay
- here, i'm creating:
- a root volume with 32 gb,
- a swap volume with 4 gb,
- a home volume occupying the remaining space.
adjust the sizes to your needs. next, create filesystems and mount everything:
mkfs.ext4 /dev/md1 # the boot partition mkfs.ext4 /dev/mapper/server-root mkfs.ext4 /dev/mapper/server-home mkswap /dev/mapper/server-swap swapon /dev/mapper/server-swap mount /dev/mapper/server-root /mnt mkdir /mnt/boot /mnt/home mount /dev/mapper/server-home /mnt/home mount /dev/md1 /mnt/boot
you can now install arch linux:
pacman -S arch-install-scripts pacstrap /mnt base genfstab -U -p /mnt >> /mnt/etc/fstab
/mnt/etc/fstab. it should have uuids this time.
arch-chroot /mnt echo your-server > /etc/hostname ln -s /usr/share/zoneinfo/Europe/Zurich /etc/localtime echo en_US.UTF-8 UTF-8 > /etc/locale.gen locale-gen pacman -Sy grub openssh screen cryptsetup sudo busybox base-devel pacman -Sy wget dropbear mkinitcpio-nfs-utils modprobe dm-mod mdadm -E --scan >> /etc/mdadm.conf
en_US with something more fitting. now check the bottom of
/etc/mdadm.conf. does it contain all three raid arrays (or how many you created)?
cryptsetup luksAddKey /dev/md2
and add a key with a long random string as a password. i'll refer to this key as
LONG_PASSWORD from now on. later, you can use this key to remotely unlock the disk on boot time. the new password should end up in slot 1. you can check the slots with:
cryptsetup luksDump /dev/md2
next, create the network configuration
/etc/netctl/wired with the same content as in the unencrypted system. then edit
/root/.ssh/authorized_keys (you might have to create
/root/.ssh first) and paste in some public ssh keys you'll want to use for login later. (we'll disable root login with password below, so you really need to do this!)
then continue with:
netctl enable wired ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key
/etc/ssh/sshd_config and add/change:
Protocol 2 HostKey /etc/ssh/ssh_host_ed25519_key HostKey /etc/ssh/ssh_host_rsa_key PermitEmptyPasswords no PermitRootLogin without-password StrictModes yes AllowUsers root Ciphers firstname.lastname@example.org,email@example.com,aes256-ctr,aes192-ctr MACs firstname.lastname@example.org,email@example.com,hmac-sha2-512,hmac-sha2-256 KexAlgorithms firstname.lastname@example.org,diffie-hellman-group-exchange-sha256
(add other user names to
AllowUsers which you will create later and want to use to remotely login via
ssh. if you're using a less-modern openssh
ssh client, or some non-openssh client, you might want to tweak the mentioned ciphers, macs and key exchange algorithms because you won't be able to connect to your server otherwise.)
remove all other host keys mentioned (or comment them out if they aren't). next, edit
/etc/ssh/moduli and remove all lines with less than 4096 bits (the fifth column contains the bitlength). these last two steps (editing files in
/etc/ssh) are not required, but do harden your system. next, run
systemctl enable sshd.service
so you can actually
ssh your new system after reboot. we now want to set up remote unlocking. (also see this document if you want to know more.) first, you should generate an rsa key for ssh communication. don't create an ecc key here, as dropbear (which we'll use) doesn't support them (you can also use
tinyssh, but that uses a different key format than
openssh, so you'll have to do some more work). on your desktop machine, run:
ssh-keygen -b 4096 -t rsa -f ~/.ssh/id_rsa_server_unlocking scp ~/.ssh/id_rsa_server_unlocking.pub root@your-server:/mnt/root/
store the private key somewhere safe; you'll need it (together with the password
LONG_PASSWORD) to remotely unlock your server. now, run the following on the server:
mkdir -p /build chgrp nobody /build chmod g+w /build cd /build for i in mkinitcpio-netconf mkinitcpio-dropbear mkinitcpio-utils; do wget https://aur.archlinux.org/cgit/aur.git/snapshot/$i.tar.gz tar -xvzf $i.tar.gz chown -R nobody:nobody $i cd $i sudo -u nobody makepkg chown root:root $i-*.xz mv $i-*.xz .. cd .. rm -rf $i done mv *.xz /root cd /root rm -rf /build cat /root/id_rsa_server_unlocking.pub > /etc/dropbear/root_key for i in mkinitcpio-netconf mkinitcpio-dropbear mkinitcpio-utils; do pacman -U $i-*.tar.xz done
- make sure everything builds and installs fine. then edit
- change the
MODULES="dm_mod dm_crypt aes_x86_64 raid1";
lvm2 mdadm_udev netconf dropbear encryptsshin the
filesystems, and add
shutdownat the end. the line should now look like
HOOKS="base udev autodetect modconf block lvm2 mdadm_udev netconf dropbear encryptssh filesystems keyboard fsck shutdown".
- change the
/usr/lib/initcpio/hooks/dropbear so that the lines starting the server look like:
echo "Starting dropbear (on port 12345)" /usr/sbin/dropbear -E -s -j -k -p 12345
i.e. add "-p 12345" to the dropbear call and printed text. this will be the port you have to connect with
ssh to to remotely unlock. you can also skip this, then you'll have to use the standard
ssh port (22).
continue with editing
/etc/default/grub. modify the
GRUB_CMDLINE_LINUX variable to
or, to be on the safe side, to
(i had trouble with the first variant some years ago). replace
xxx.xxx.xxx.xxx with your server's ip,
yyy.yyy.yyy.yyy with the gateway,
zzz.zzz.zzz.zzz with the hostmask and
your-server with your server's hostname. you also might have to adjust
eth0 in case your server has more than one network interface. (for me,
eth0 always worked.)
next, set a root password, create the init ramdisk, and set up the boot loader:
passwd mkinitcpio -p linux grub-install --recheck /dev/sda grub-mkconfig -o /boot/grub/grub.cfg
in case you're using a old mbr partition table, you might have to set the bootable flag for the boot partition.
then, exit the chroot and reboot:
exit umount -R /mnt swapoff /dev/mapper/server-swap cryptsetup luksClose cryptdisk sync reboot
unlocking the encrypted arch linux installation
your server should now boot into the init ramdisk, start dropbear, and wait for a connection to unlock your encrypted partition. to unlock it, run:
echo LONG_PASSWORD | ssh -p 12345 -i ~/.ssh/id_rsa_server_unlocking root@your-server
this should unlock your encrypted disk (which takes around 10 seconds if you followed my steps to the letter), and then boot arch linux. you'll be able to log in as
root with the ssh keys you inserted earlier. form that point on, you can configure the system like any random linux installation via
ssh (for example, by using ansible).
if the system doesn't come up or doesn't start networking (you can use
ping to see whether the network interface is up; as soon as it responds to ping after reboot, you can try the above
ssh unlocking command), you can either reboot into your hoster's rescue system, mount and chroot the unencrypted system, and rewrite the boot loader to reboot in the unencrypted system, and/or use a serial console and/or kvm to find out what went wrong. anyway, debugging such a situation is really hard, so good luck! but if your system is close enough to mine and you followed the above steps correctly (and i didn't screwed something up), it should work.
this year, we also visisted oslo. we were lucky, and the weather was superb! sun all the time, it was really beautiful.
we did some excursions, namely to ulvøya in the south-east part of oslo, …
… to nesoddtangen, …
finally, some shots of tjuvholmen by night:
anyway. setting it up went smooth, especially with the instructions from that blog post. only for fontein.de it was a bit more tricky, since i also had to add the public key as otherwise denic didn’t like the record.
now the only thing missing is that cablecom actually provides a dnssec-capable dns resolver…
it finally started snowing. well, already two days ago. but today, for the first time, it was sunny again! (and we started moving the snow away from the balcony and the pathways so the cats won’t take a dump there…)
this is how it looks like from out of a cat tree:
once during the last days of december, a great fog panorama presented itself to us. the fog was close, but didn’t reach us yet:
in the last 30 days, a lot of things happened. the worst thing happening was that one of our cats, töffel, lost more and more strength and started looking really miserable. since in the weeks before we already had to give him infusions as he was drinking way too little, and he hated that pretty much, we decided not to start more medicine experiments and instead to stop his suffering by putting him to sleep. he’s probably better off now.
rest in peace, töffel!
this is awesome!
for one, this allows me to get some “real” certificates (as opposed to my self-signed ones) without paying a larger sum of money per year (i’m using quite many subdomains of fontein.de and two other domains, which results in quite some sum even when using cheap resellers of resellers of resellers).
then, their goal is to automate the whole process as much as possible. so instead of a lof of manual work (mostly filling out forms, handling payment of fees, reacting to emails or domain challenge requests, etc.) it should be possible to run one command, maybe even as a cronjob, to get a (renewed) certificate for a domain or a set of domains.
that’s already much better, but still not what i want, as this is hard to automate when you don’t want to run that on the webserver itself. i’m prefering something which can run somewhere else, and can be integrated in an orchestration tool like ansible. well, so i took daniel roesler’s code (including a python 3 patch by collin anderson) and converted it into a more modular tool, which allows to split up the process so that with some more scripting, it can easily be used to do the process from remote. you can find the result on github. i also created an ansible role which allows to simply generate keys, certificate signing requests and get complete certificates from let’s encrypt with ansible; that project can also be found on github. i’m using it in production for my personal webserver: as a result you can now look at spielwiese without having to accept my self-signed certificate! maybe also others will find this useful.