skip to main content.

posts for april 2014.

yesterday, i was in our garden, doing a bit of work. namely, planting some lavender. two of the plants you can see here:

i haven’t planted anything (nor did any serious garden work, in fact) for quite some years now, but it was fun. (and i wasn’t forced to do it, either.)

the main purpose of the lavender is although not to just be a nice view – that’s only the secondary motive – but to make people realize that they shouldn’t walk through our garden and treat it as a kind of short-cut to get onto the hill. i hope this helps, and that not at some point some idiot just stamps my lavender to death.

(well, currently it also looks a bit broken; that was thanks to the heavy rain this night. i hope it will recover from it and hopefully start growing soon.)

last summer, after buying a new four terabyte harddisk for my main computer (replacing the old and notoriously full one terabyte harddisk), i wanted to try something new. instead of using ext2/3/4, i decided to switch to the btrfs filesystem. the main feature why i wanted to use btrfs was the ability to quickly create snapshots of the current disk content on the fly, thus being able to browse through how the disk looked some time ago. the snapshots are essentially only the difference between the old data and the new, thus they are essentially free if the disk content isn’t changing a lot between the snapshots. which, at least for me, is usually the case.
i’m using btrfs only for the /home partition, to which i added a subdirectory /home/backup to store backups. in this post, i want to explain how to set up a simple system which makes a snapshot every ten minutes, and cleans up older snapshots so that

  • for snapshots older than a day, only one snapshot is left for every hour, and
  • for snapshots older than a week, only one snapshot is left for every day, and
  • for snapshots older than a year, only one snapshot is left for every month.

so even with a lot of changes inbetween, the number of snapshots shouldn’t be too big, and thus not too much space will be wasted, while still allowing to access old (and deleted!) data. note that changing the interval from every ten to, say, every minute should be no problem. if you ever accidently delete something, you’ll have no problem to resurrect the file even if you only notice some hours, days, weeks or even months later. (providing that the file has already been around for at least a similar time interval.)

one note regarding btrfs in general. while btrfs is still marked experimental, it seems to be pretty stable in practice. the only caveat is that you should never fill btrfs disks too much. always make sure enough space is left. that shouldn’t be a problem for my four terabyte disk for quite some time, but in case you love to quickly fill space, better get more than one drive and join them (via raid zero or something like that). also, note that one btrfs filesystem can span over several partitions and disks, and that it can internally do several raid modes. in fact, that’s something i want to try out soon, by combining a bunch of older harddisks i’ve still lying around in a jbod array and putting a raid one btrfs filesystem over all of them. note that btrfs will in the future allow to configure this even more refined (like increasing redundancy, or also using different configurations per file), and that it’s always possible to update a filesystem on the fly while it is mounted.

creating read-only snapshots.

creating a read-only snapshot is simple: just run btrfs subvolume snapshot -r /home /home/backup/name_of_snapshot. (if you want snapshots you can also write to, drop the -r.) for example, you could create a little shell script:

1#!/bin/bash
2TIMESTAMP=`date +"%Y-%m-%d-%H%M%S"`
3btrfs subvolume snapshot -r /home /home/backup/$TIMESTAMP
4rm -rf /home/backup/$TIMESTAMP/backup/20*

this creates a read-only snapshot based on the current date, and cleans up the /backup subdirectory of /home/backup in the snapshot. after all, we don’t want to recursively increase the tree’s depth by having links to all older snapshots in each snapshot.

setting up your computer to execute this script regularly is quite simple. let’s say it is stored as /home/backup/snapshot.sh with read and execution priviledges for root; then you could run crontab -e as root and add a line like
1,11,21,31,41,51 * * * * root /bin/bash -c "/home/backup/snapshot.sh &>> /var/log/snapshot.log"
this runs the script at xx:01, xx:11, xx:21, xx:31, xx:41 and xx:51 for every hour xx on every day during the whole year. the script’s output (which should be essentially something like Create a snapshot of '/home' in '/home/backup/2014-04-27-000100') is stored in a log file /var/log/snapshot.log.

cleaning up.

cleaning up is a little more complicated. deleting a snapshot itself is easy: just run btrfs subvolume delete /home/backup/name_of_snapshot. to delete snapshots according to the rules i wrote up above, i wrote a little python script:

 1#!/usr/bin/python2
 2import os, os.path, datetime, subprocess
 3
 4class CannotParse(Exception):
 5    pass
 6
 7# Find all directories in /home/backup
 8now = datetime.datetime.now()
 9td_day = datetime.timedelta(days=1)
10td_week = datetime.timedelta(weeks=1)
11td_month = datetime.timedelta(days=31)
12monthold = dict()
13weekold = dict()
14dayold = dict()
15rest = dict()
16for file in os.listdir('/home/backup'):
17    if not os.path.isfile(file):
18        # Interpret name as timestamp
19        data = file.split('-')
20        try:
21            if len(data) == 4:
22                year = int(data[0])
23                month = int(data[1])
24                day = int(data[2])
25                if len(data[3]) == 4:
26                    hour = int(data[3][0:2])
27                    minute = int(data[3][2:4])
28                    second = 0
29                elif len(data[3]) == 6:
30                    hour = int(data[3][0:2])
31                    minute = int(data[3][2:4])
32                    second = int(data[3][4:6])
33                else:
34                    raise CannotParse()
35                timestamp = datetime.datetime(year, month, day, hour, minute, second)
36                isodate = timestamp.isocalendar() + (hour, minute, second)
37            else:
38                raise CannotParse()
39            
40            age = now - timestamp
41            if age >= td_month:
42                id = isodate[0:2]
43                d = monthold
44            elif age >= td_week:
45                id = isodate[0:3]
46                d = weekold
47            elif age >= td_day:
48                id = isodate[0:4]
49                d = dayold
50            else:
51                id = isodate[0:6]
52                d = rest
53            if id not in d:
54                d[id] = list()
55            d[id].append([timestamp, file])
56        except Exception:
57            pass
58
59def work(d, title):
60    for id in d:
61        list = d[id]
62        list.sort()
63        if len(list) > 1:
64            for v in list[1:]:
65                retcode = subprocess.call(['btrfs', 'subvolume', 'delete', '/home/backup/' + str(v[1])])
66                if retcode != 0:
67                    print 'Error! (Return code ' + str(retcode) + ')'
68
69work(monthold, "MONTH OLD:")
70work(weekold, "WEEK OLD:")
71work(dayold, "DAY OLD:")
72work(rest, "REST:")

i stored it as /home/backup/cleanup.py and made it runnable by root, and scheduled it to be run every hour at a fixed minute offset (say, xx:59) by running crontab -e and adding
59 * * * * root /bin/bash -c "/home/backup/cleanup.py &>> /var/log/snapshot.log"
again, the output is put into /var/log/snapshot.log.

posted in: computer
tags:
places:

yesterday, we were in winterthur, watching cyclope, a wonderful show combining tinguely-like apparatuses with acrobatics and music. it was really amazing! the show, inspired by jean tinguely’s le cyclop, is performed in an old industrial hall, which creates a wonderful atmosphere. the stage, which begins directly in front of the persons sitting in the first row, is a broken up amusement park, which seems to be only inhabited by a (involuntary?) clown. when he’s eventually fed up by his surroundings and wants to leave, suddenly hell breaks loose. it was just amazing! a wide range of acrobatics, acting and music blending perfectly in the huge metal colossus, which turns more and more into a face with, well, only one eye. a pretty heavy piece of metal, so to say.

we were lucky, and our friends, who arrived very early, were offered an upgrade to sit more in the front, and so we ended up in row 9 (instead of something much higher). it looks like it wasn’t exactly sold out, and they tried to at least fill the first rows so that it doesn’t look too sad. which is really, really shameful, since such a great show deserves a much larger audience! so in case you’re somewhat close to winterthur, be sure to watch cyclope while it is still shown! (until end of may, apparently.) after that, it apparently will be shown in july in basel as well (though outdoors there). it is really something which should be seen live, and not on video or tv or anything similar.