i just read that finally, after many years of waiting, the new c++ iso standard was approved, now labelled c++11!

i thought a lot about social networks recently. i want to write some of these down here.

introduction.

during the last few decades, computers and internet made deep changes to our society, to how we communicate, to how we live. electronic communication has existed for a very long time, starting with bulletin board systems (starting in the 1970s), usenet (starting 1980) and internet relay chat (starting 1990). later, after the advent of the internet to public (in form of the world wide web) around 1995, new services emerged, like the ICQ instant messaging service and sixdegrees.com, one of the earliest “modern” social networks. later, myspace became well-known, as well as business-oriented networks such as linkedin, and later facebook (starting in the united states, later going worldwide) and studivz (started in germany and german-speaking countries).

facebook is the most known representant nowadays, but there are many more social networks out there – the wikipedia list for example contains 709 entries. depending on in which area you live, some social networks are much more popular amoung your local friends than others. (an article about a study analyzing the situation in germany some time ago can be read here.)

social networks play a more and more important role. you need to be connected to your friends to find out what’s going on. to see who’s going with whom, who’s a “friend” of whom, what’s in and what not, where and when are the parties. this does not only applies to the young generation anymore, especially not to just a small subset, but to a large part of society. it is not uncommon in some areas that even your parents or grandparents are on social networks. social networks allow you to see what happens to your family, friends, to people you know but don’t have much contact with anymore. you find out who of your high school friends is marrying whom, you see photos from vacations of people you haven’t seen since kindergarten, you find out that someone you knew from university got a job, or find out that some guy you met on a vacation ten years ago now became father. a lot of these things you would have missed without a social network, maybe found out about later by chance, but more probably never heard about them at all.

so definitely, social networks play an important role.

criticism.

there are two fundamentally different criticisms one can write about.

the first one is about on the change of privacy, on the extended focus. things you say, you do, are now not just noted (and more or less quickly forgotten) by the people being present at that moment, but often dragged into your social network, shared with all your friends there, which might include distant friends you met at kindergarten, colleagues at work, many people with whom you are studying, your neighbor, your extended family, or whoever else can see what happens on your profile. you do something stupid, someone takes a photo of it, puts it into the network, tags you and everyone can see what you did. you wrote something stupid or very private, accidently on someone’s wall instead in a private message, and suddenly many people know about it. and not only that, depending on the privacy settings of the social networks, maybe the whole internet can read or see these things. but i don’t want to write about these topics today.

the other big problem, from my point of view, is the data ownership. think about it. why should a company create such a social network? provide lots of computing power to allow people to communicate, to search for friends, to exchange photos, etc., and that essentially for free? companies want to make money. in fact, need to make money, to pay for the servers, for the programmers, for the support people. of course, there are ads, which make some of the money. without ads it is essentially impossible to run a huge network. but ads are not everything. what is also very important is the collection of information. information on people, their age, gender, preferences, interests, friends, what they like or not, what they find interesting. if the state would try to get this information, people would protest against it. but on the internet, they give it to a company essentially for free. of course, it is true that many of these information pieces are available on the net anyway, at least for people like me. but then, if you have to collect them yourself, this costs a lot of time. if i have a profile at some social network and enter everything there into a form, they get all the information in a well-defined format which can easily be processed.

consider for example facebook. if you have a facebook account, they usually know your name, birthdate, email adress, gender, sexual interest, where you live, work, what your marital status is, who your friends are, which websites you like. some people also use facebook as their search machine, so facebook also knows what you search for. and depending on how websites included the facebook “like” button, facebook knows which websites you visit. if you’re logged in at the same time, they can combine that information with your profile to see what you’re doing on the web. since some time, facebook also tries to find out your location, by encouraging you to tell it to them, and also tell your friends’ locations to them. so they can also track you in the real world. besides these things, facebook is also known (and often criticized) for their very liberal view of privacy, and for storing all information without really allowing to delete it.

or consider google+. if you have an account there, google knows your personal information such as name, email adress, birthdate, … but besides that, google knows much more about you. google is the number one search engine in many parts of the world, and so most people use it to search for something. if you use their search engine while you are logged in at google+, they can connect that information. moreover, google analytics is a free service aimed at website administrators, which allows them to see how many people look at their website, what they do there, where they come from, etc. but it also allows google to see what people do. and if you have a google account (not just google+!), they can actually see what you are doing on the web. a huge amount of websites uses some google service or another. many google services are included by using some javascript, which is loaded from a google server, and so google can see where you are on the web and what you are doing there.

think about it. if the state would send out secret agents which would follow any person, look at what they do, where they are at any moment, what they look at. like in 1984. would you like that? i guess, most of you wouldn’t. but yet, many people allow google and/or facebook to do exactly that, without spending a thought about it.

a possible solution.

so now what? should one simply try not to use facebook or google? stick to smaller social networks, smaller services, which cannot track you that well? especially using smaller social networks would destroy a lot: many of your friends or people you know might not be in your network anymore, maybe forcing you to have accounts for many different social networks. this would make life much more complicated (which people do not want to), and is in practice just annoying. so this is not a solution.

if one wants to use social networks at all, one does not want such fragmentation. but one also does not want certain entities, such as big corporations or even the state, to collect all that information at one place. so the best solution would be to distribute the information in some way, splitting it up so that single entities such as google or facebook or the state cannot access most of it, but you can still see information about your friends, still contact them, communicate with them.

there is in fact a social network designed like this: diaspora. everyone can run their own diaspora server, you can have friends on any other diaspora server, you see what they do, you can communicate. every server can only see what’s going on the server, and what’s going on with the friends of the people having an account on that server, as far as these friends allow the people with an account on this server to see their actions.

unfortunately, when the first alpha version of diaspora was released, it had many vital security problems, making it essentially not useable for anyone with the slightest sensitivity for privacy and security. i don’t know what the current status is, i hope it dramatically increased. but even though the reference implementation is not good, everyone can create their own implementation, which could then communicate with servers running the reference implementation, or also any other diaspora implementation. this is what makes diaspora very attractive: you are not forced to use or trust specific implementations and servers. still, diaspora is probably far from perfect. i guess that one could write books about how to design a very good open social network. i wouldn’t be surprised if there are even research projects working on such topics.

anyway. in my opinion, the future belongs to such open distributed social networks. as soon as such a network becomes useable enough, i will migrate to it.

(originally, i wanted to discuss properties of such open distributed social networks in more details, discuss which aspects are important, discuss security of information, etc. but i’m afraid if i would really do this, the result would be way too long for a single blog post. and it will take a lot of time to write this down in detail and to reach a good enough description and discussion of most aspects; time which i simply don’t really have.)

(and a note about myself: i’ve been using several different social networks in the past, most prominently facebook and studivz. except facebook, i’ve deleted all my accounts. i’d also like to delete the facebook account, since i don’t really trust and like facebook, but the lack of alternatives currently makes me staying there. i haven’t tried diaspora yet, but that’s still on my to-do-list, though i want to wait until that projects reaches a more stable state.)

i recently presented a bash script which schedules computational tasks on multi-core machines. in the meanwhile, i fixed a bug in the display, made the program more flexible, and started to use local variables instead of global variables only. the new version is also more intelligent: it tries to adjust the running times of its controlled processes so that the running times are not far apart.
here is the newest version:

  1#/bin/bash
  2
  3initProfile() {
  4    PROFILEFN=bigprimerunner-$PROFILE.profile
  5    CORES=`grep "^CORES " $PROFILEFN`
  6    CORES=${CORES/CORES }
  7    STARTUP=`grep "^STARTUP " $PROFILEFN`
  8    STARTUP=${STARTUP/STARTUP }
  9    eval STARTUP=$STARTUP
 10}
 11
 12# Startup
 13LOADMODIFIER=0
 14if [ "$1" != "" ]
 15then
 16    PROFILE=$1
 17else
 18    PROFILE=`hostname`
 19fi
 20if [ "$2" != "" ]
 21then
 22    LOADMODIFIER=$2
 23fi
 24initProfile
 25if [ "$CORES" == "" ]
 26then
 27    echo "Cannot load profile $PROFILEFN!"
 28    exit
 29fi
 30echo Cores: $CORES
 31echo Load modifier: $LOADMODIFIER
 32
 33# The command to execute
 34COMMAND=primefinder
 35
 36computeFreecores() {
 37    FREECORES=0
 38    local DAY=`date +%w`
 39    local LINE=`grep "^$DAY " $PROFILEFN`
 40    local LINE=${LINE/$DAY }
 41    local HOUR=`date +%k`
 42    for ((i=0;i<$HOUR;++i));
 43    do
 44        local LINE=${LINE#* }
 45    done
 46    local LINE=${LINE/ *}
 47    eval FREECORES=$LINE
 48    # Also determine how many jobs should be started
 49    STARTUP=`grep "^STARTUP " $PROFILEFN`
 50    STARTUP=${STARTUP/STARTUP }
 51    eval STARTUP=$STARTUP
 52}
 53
 54killProcess() { # One argument: PID of process to kill
 55    local PID=$1
 56    local FILE=`lsof -p $PID -F n 2>/dev/null | grep primedatabase | grep -v "\.nfs"`
 57    kill $PID 2> /dev/null
 58    local A=${FILE#n*}
 59    local A=${A/ (nfs*}
 60    if [ "$A" != "" ]
 61    then
 62        rm $A
 63        echo Killed $PID with open file $A
 64    else
 65        echo Killed $PID with no open file
 66    fi
 67}
 68
 69stopsignal() {
 70    local PIDS=`jobs -p`
 71    echo
 72    echo
 73    echo Terminating...
 74    echo Killing: $PIDS
 75    for PID in $PIDS;
 76    do
 77        killProcess $PID
 78    done
 79    echo done.
 80    exit
 81}
 82
 83trap 'stopsignal' 2
 84
 85computeFreecores
 86
 87echo "Starting $STARTUP instances (in $BINDIR)"
 88
 89filterRunning() { # Removes all PIDs from the arguments which are currently stopped
 90    ps -o pid= -o s= $* | grep R | sed -e "s/R//"
 91}
 92
 93filterStopped() { # Removes all PIDs from the arguments
 94    ps -o pid= -o s= $* | grep T | sed -e "s/T//"
 95}
 96
 97determineToAdd() {
 98    computeFreecores
 99    local LOAD=`uptime`
100    local LOAD=${LOAD#*average: }
101    local LOAD=${LOAD/,*}
102    local LOAD=${LOAD/.*}
103    ADD=$[CORES-FREECORES-(LOAD+LOADMODIFIER)]
104    local JOBS=`jobs -p`
105    local JOBS=`filterRunning $JOBS`
106    echo "Load: $[LOAD+LOADMODIFIER], Intended number of free cores: $FREECORES, Running: `echo $JOBS | wc -w`, Started: `jobs -p | wc -l` (should be $STARTUP)"
107}
108
109continueOne() {
110    local JOBS=`jobs -p`
111    local JOBS=`filterStopped $JOBS`
112    if [ "$JOBS" != "" ]
113    then
114        local PID=`ps -o pid= --sort +time $JOBS | head -1`
115        echo Continuing $PID...
116        kill -SIGCONT $PID
117    fi
118}
119
120stopOne() {
121    local JOBS=`jobs -p`
122    local JOBS=`filterRunning $JOBS`
123    if [ "$JOBS" != "" ]
124    then
125        local PID=`ps -o pid= --sort -time $JOBS | head -1`
126        echo Stopping $PID...
127        kill -SIGSTOP $PID
128    fi
129}
130
131killOne() {
132    local JOBS=`jobs -p`
133    if [ "$JOBS" != "" ]
134    then
135        local PID=`ps -o pid= --sort -time $JOBS | head -1`
136        killProcess $PID
137    fi
138}
139
140launchOne() {
141    echo "Launching \"$COMMAND\"..."
142    $COMMAND &
143    sleep 1.5
144}
145
146computeTotaltimeInSecs() {
147    # Input: $1
148    # Output: $TOTALSECS
149    local I=$1
150    local SECS=${I##*:}
151    local REST=${I%:*}
152    local MINS=${REST##*:}
153    local REST=${REST%:*}
154    local HOURS=${REST##*-}
155    local DAYS=`expr "$REST" : '\([0-9]*-\)'`
156    local DAYS=${DAYS%-}
157    if [ "$DAYS" == "" ]
158    then
159        local DAYS=0
160    fi
161    if [ "$HOURS" == "" ]
162    then
163        local HOURS=0
164    fi
165    if [ "$MINS" == "" ]
166    then
167        local MINS=0
168    fi
169    echo "((($DAYS * 24) + $HOURS) * 60 + $MINS) * 60 + $SECS" | bc
170}
171
172adjustProcesses() {
173    local JOBS=`jobs -p`
174    local JOBS=`filterRunning $JOBS`
175    if [ "$JOBS" != "" ]
176    then
177        local STOPPID=`ps -o pid= --sort -time $JOBS | head -1`
178        local JOBS=`jobs -p`
179        local JOBS=`filterStopped $JOBS`
180        if [ "$JOBS" != "" ]
181        then
182            local CONTPID=`ps -o pid= --sort +time $JOBS | head -1`
183            # Compute times
184            local I=`ps -o time= $STOPPID`
185            local STOPSEC=`computeTotaltimeInSecs $I`
186            local I=`ps -o time= $CONTPID`
187            local CONTSEC=`computeTotaltimeInSecs $I`
188            # Compare times
189            local CT=`echo $CONTSEC+60*5 | bc`
190            if [ $STOPSEC -gt $CT ]
191            then
192                echo Stopping $STOPPID and continuing $CONTPID
193                kill -SIGSTOP $STOPPID
194                kill -SIGCONT $CONTPID
195            fi
196        fi
197    fi
198}
199
200# Start programs in the background
201determineToAdd
202for ((i=1;i<=STARTUP;++i));
203do
204    launchOne
205    if [ $i -gt $ADD ]
206    then
207        sleep 1
208        kill -SIGSTOP %$i
209    fi
210done
211
212# Start mainloop
213while [ 1 ]
214do
215    sleep 60
216    
217    # Determine how many processes should be added/removed
218    determineToAdd
219
220    # Stop/continue processes
221    if [ $ADD -gt 0 ]
222    then
223        # Add processes
224        echo ADD:$ADD
225        for ((i=0;i<ADD;++i))
226        do
227            continueOne
228        done
229    fi
230    if [ $ADD -lt 0 ]
231    then
232        REM=$[-ADD]
233        # Remove processes
234        echo REMOVE:$REM
235        for ((i=0;i<REM;++i))
236        do
237            stopOne
238        done;
239    fi
240
241    # Launch new processes or kill running ones
242    CURRLAUNCHED=`jobs -p | wc -l`
243    if [ $STARTUP != $CURRLAUNCHED ]
244    then
245        if [ $STARTUP -lt $CURRLAUNCHED ]
246        then
247            echo kill: $STARTUP $CURRLAUNCHED
248            for ((i=STARTUP;i<CURRLAUNCHED;++i));
249            do
250                killOne
251            done;
252        else
253            echo add: $CURRLAUNCHED $STARTUP
254            for ((i=CURRLAUNCHED;i<STARTUP;++i));
255            do
256                launchOne
257            done;
258        fi
259    fi
260    sleep 2
261    
262    # Adjust
263    adjustProcesses
264done

ever had the problem that you have access to a big machine (with many cores), and you want to run many (tens of thousands) small computations, but you want to make sure that not too many cores are used?
i’ve had this problem, and since i now have a pretty nice (i think so) solution, i thought that maybe more people are interested in it. so here’s my setup. i have a program, let’s call it primefinder, which, for a certain input n (where n is a natural number ≤ 21000), computes a prime of n bits with special properties. the program loops over all possible n, and checks for each n if a file n.prime exists. if it does not, it creates it (with zero content), computes the prime (which can take between minutes and days), writes the prime into the file and continues with the next file. this simple task distribution technique allows me to run the program in parallel on different machines (since the files are in a nfs folder) with many instances on each machine. now at our institute, we have a big computation machine (64 cores) and four user machines (on which the users work, each 32 cores). since the user machines are often not intensively used (and that only during certain times of the day), i want to use these as well. but there should be enough cores free, so the users won’t notice that there are computations going on in the background. on the computation server, also other people want to run something, so there should also be some free cores. optimally, my program would somehow decide how many cores are used by others, and use the rest. or most of them, to leave some free, especially on the user machines.
after a suggestion by our it guys, i started writing a bash script which controls the instances of my program on the same machine. the first version used the time of the day to determine the number of processes. everything was computed in terms of the number of cores of the machine, the load (with a load modifier applied, since some machines have uninterruptable processes running which do not effectively do something, and which won’t go away until the next reboot) and the hour of the day. but it is not easy to find a good scheme which yields good results on all machines. something which works well on the user machines is wasting processor time on the computation server.
so today i rewrote the program to use profiles. a profile contains information on the number of cores (this is necessary since the computation server has hyperthreading enabled, and thus returns twice the number of cores), the number of processes to be started, and the number of cores to be left free during each hour and day of a week. so on weekends or nights, i choose lower numbers for the free cores for the user machines, while for the computational server the number is always 1.
a profile can look like this (this is from a user machine, the file is called primefinderrunner-user.profile for later reference):

1CORES 32
2STARTUP $[CORES-CORES/8]
30 $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8]
41 $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/4] $[CORES/4] $[CORES/4] $[CORES/8]
52 $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/4] $[CORES/4] $[CORES/4] $[CORES/8]
63 $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/4] $[CORES/4] $[CORES/4] $[CORES/8]
74 $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/4] $[CORES/4] $[CORES/4] $[CORES/8]
85 $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/2] $[CORES/4] $[CORES/4] $[CORES/4] $[CORES/8]
96 $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/8] $[CORES/16] $[CORES/16] $[CORES/16] $[CORES/16]

the line with prefix CORES gives the number of cores. the line prefixed by STARTUP gives the number of processes to run (at most); here, we use 7/8 of the number of cores. the lines prefixed by a number between 0 (sunday) and 6 (saturday) have 24 entries following: every entry (seperated by exactly one space, as the prefix itself is separated by exactly one space from the entries!) says how many cores should be free at each time of the day. usually during night (up to 7 am) at least 1/16 of the total number of cores should be free, while during workday (8 am to 7 pm) half of the cores should be free. of course, the numbers are different for weekends (saturday and sunday) than for the other working days.
now the script itself looks like this (for reference, the filename is primefinderrunner.sh):

  1#/bin/bash
  2 
  3initProfile() {
  4    PROFILEFN=primefinderrunner-$PROFILE.profile
  5    CORES=`grep "^CORES " $PROFILEFN`
  6    CORES=${CORES/CORES }
  7    STARTUP=`grep "^STARTUP " $PROFILEFN`
  8    STARTUP=${STARTUP/STARTUP }
  9    eval STARTUP=$STARTUP
 10}
 11 
 12LOADMODIFIER=0
 13if [ "$1" != "" ]
 14then
 15    PROFILE=$1
 16else
 17    PROFILE=`hostname`
 18fi
 19if [ "$2" != "" ]
 20then
 21    LOADMODIFIER=$2
 22fi
 23initProfile
 24if [ "$CORES" == "" ]
 25then
 26    echo "Cannot load profile $PROFILEFN!"
 27    exit
 28fi
 29echo Cores: $CORES
 30echo Load modifier: $LOADMODIFIER
 31 
 32computeFreecores() { 
 33    # two arguments: day (0..6) and hour (0..23)
 34    FREECORES=0
 35    DAY=`date +%w`
 36    LINE=`grep "^$DAY " $PROFILEFN`
 37    LINE=${LINE/$DAY }
 38    HOUR=`date +%k`
 39    for ((i=0;i<$HOUR;++i));
 40    do
 41        LINE=${LINE#* }
 42    done
 43    LINE=${LINE/ *}
 44    eval FREECORES=$LINE
 45}
 46 
 47computeFreecores
 48 
 49stopsignal() {
 50    for PID in `jobs -p`;
 51    do
 52        FILE=`lsof -p $PID -F n 2>/dev/null | grep primedatabase | grep -v "\\.nfs"`
 53        A=${FILE#n*}
 54        A=${A/ (nfs*}
 55        echo killing $PID with open file $A
 56        rm $A
 57        kill $PID
 58    done
 59    exit
 60}
 61 
 62trap 'stopsignal' 2
 63 
 64echo "Starting $STARTUP instances"
 65 
 66determineToAdd() {
 67    computeFreecores
 68    LOAD=`uptime`
 69    LOAD=${LOAD#*average: }
 70    LOAD=${LOAD/,*}
 71    LOAD=${LOAD/.*}
 72    ADD=$[CORES-FREECORES-LOAD-LOADMODIFIER]
 73    echo Load: $[LOAD-LOADMODIFIER], Intended number of free cores: $FREECORES
 74}
 75 
 76# Start programs in the background
 77determineToAdd
 78for ((i=1;i<=STARTUP;++i));
 79do
 80    primefinder &amp;
 81    sleep 2
 82done
 83sleep 20
 84if [ $ADD -lt 0 ]
 85then
 86    ADD=0
 87fi
 88for ((i=ADD+1;i<=STARTUP;++i));
 89do
 90    kill -SIGSTOP %$i
 91done
 92 
 93CURRRUNNING=$ADD
 94RUNNINGSTART=1 # The first one running
 95RUNNINGSTOP=$CURRRUNNING # The last one running
 96 
 97startOne() {
 98    # Assume that $CURRRUNNING < $STARTUP
 99    RUNNINGSTOP=$[(RUNNINGSTOP % STARTUP) + 1]
100    kill -SIGCONT %$RUNNINGSTOP
101    CURRRUNNING=$[CURRRUNNING+1]
102}
103 
104stopOne() {
105    # Assume that $CURRRUNNING > 0
106    kill -SIGSTOP %$RUNNINGSTART
107    RUNNINGSTART=$[(RUNNINGSTART % STARTUP) + 1]
108    CURRRUNNING=$[CURRRUNNING-1]
109}
110 
111# Start mainloop
112while [ 1 ]
113do
114    sleep 60
115 
116    # Determine how many threads should be added/removed
117    determineToAdd
118    if [ $ADD -gt 0 ]
119    then
120        if [ $[ADD+CURRRUNNING] -gt $STARTUP ]
121        then
122            ADD=$[STARTUP-CURRRUNNING]
123        fi
124        # Add processes
125        echo ADD:$ADD
126        for ((i=0;i<ADD;++i))
127        do
128            startOne
129        done
130    fi
131    if [ $ADD -lt 0 ]
132    then
133        REM=$[-ADD]
134        # Clip
135        if [ $REM -gt $CURRRUNNING ]
136        then
137            REM=$CURRRUNNING
138        fi
139        # Remove processes
140        echo REMOVE:$REM
141        for ((i=0;i<REM;++i))
142        do
143            stopOne
144        done
145    fi
146    sleep 60
147done

the script first starts all instances, then stops the ones which are too many, and then starts the main loop. in the main loop, it waits 60 seconds (for the average load to adjust to the new process count), and then decides how many cores should be left free, and what that means for the number of processes (add/remove some). note that the profile file is read every minute, so it can be changed any time without any need to re-run the whole thing.
in case the script is stopped (with control+c), all primefinder processes are killed and their open file is deleted. to determine the open file, i use lsof with some greps. you have to adjust and test that line before using this script!
note that this script is quite a hack, and far from perfect. and it is somehow system dependent, or at least “setup dependent” since it has certain assumptions on the executables, on how the output of lsof looks like, … so better make sure it works before you use it, especially on bigger systems. also note that in the beginning, all instances are ran (they are started with a two second delay between two instances), and then everything is run for 20 seconds before the first adjustment (i.e. stopping processes which are too many) are made. if you share the system with other people, this might already annoy others when they try to measure timings of their programs (especially if hyperthreading is enabled).

today i had a problem with latex. i’m working on graphical output of computational results, and created graphical representation of certain large matrices using libpng, and wanted to include these graphics into the automatically generated .tex files which present the results. it turned out that latex got stuck when image file names such as 1.4-0.362578-10.transform.png turn up, complaining that 4-0.362578-10.transform.png is an unknown file extension. after doing a little research, i found the grffile latex package. including it solved the problem right away… maybe some more people find this useful.

one of the video/animation compo entries at revision was raum/zeit by subdream. it displays a lot of three-dimensional fractals and scored the second place. this was one of my favorite entries for that compo, and so here it is:

another fractal based compo entry was electronenmultiplizierer by akronyme analogiker, which is a 4k intro, i.e. an executable of size four kilobytes (that is a very small amount!) which plays something in realtime. you can watch a low-quality recording here:

this weekend, i attended revision, a demoparty. revision is the successor of breakpoint, which in turn is the successor of mekka & symphosium. these events were held over easter from 1997 on, and i attended a few of them (between 2001 and 2005).
after the end of the breakpoint era last year, the revision was moved from bingen to saarbrücken, germany. a city i haven’t been in for quite some time. the party was held in the e-werk, an old power plant.

the weather was really good, it was sunny most of the time except a thunderstorm one evening and a few rain drops. this time, i stayed in a hotel and didn’t sleep under the tables, which resulted in much more sleep for me, except that breakfast at the hotel was pretty early compared to the times i usually ended up in bed asleep. but at least i got decent sleep and didn’t look like a zombie after two days.
it was really nice to meet some friends again, most of them after a long time, in particular trigger, kojote, t$, dynamite, and some more.

the following photos show a bit of what was happening. the first and third photo shows people watching a compo (unfortunatley you cannot see anything on the bigscreen, otherwise everything else would have not been really visible), and the fourth shows the underground hallway to the restrooms.

one afternoon i went on a walk with kojote, looking at some interesting buildings in the neighborhood. here are some impressions:

overall, i have to say that it was partially exhausting, but nonetheless another nice experience. the next time i’ll just stay some of the days, i guess. maybe i’ll drop by at the buenzli this year, since it is not very far from here.

does anyone knows how the ulimit bash (or whatever shell you like) command works? i’m currently running a few instances of the same program on a big server, and it happened yesterday twice that the programs together ate up all available memory (some of the instances using much more than others). there was a hard ulimit set on memory, and the result was that all these processed were killed. not just the one violating the memory limit in that moment.
this sucks pretty much, since this destroyed some cpu days of work. does anyone knows why ulimit is doing this? i assume that the rationale is to stop fork bombs, but in this case this is really, really annoying. killing one of the processes would have been perfectly enough…
so, if anyone has good documentation on how ulimit works, whether it is possible to change this behaviour, and whether this is actually intended or a bug, i would like to hear about it…

in the last weeks, i had to compile several libraries for our ultrasparc machine running solaris (sunos 5.10). in particular, these libraries were gmp, atlas, iml, ntl and boost. i wanted to use the sun studio c/c++ compiler (cc has version 5.8, CC has version 5.9) instead of gcc/g++. moreover, i need 64 bit versions of everything, since my programs need a lot of memory. (the machine has around 140 gb of ram anyway, so it makes a lot of sense.)

since it was somewhat troublesome to get everything running (atleast running enough so that i could use what i needed), i want to describe the process of compiling everything here. maybe this is useful for someone…

i compile everything into my home directory, /home/felix. i also use stlport4 instead of the sun studio standard c++ stl, since i couldn’t figure out how to compile boost with the usual stl. the code generated will not be portable, but should be fast.

gmp.

for configuration and compilation, i did the following:

$ export CC=cc
$ export CXX=CC
$ export CFLAGS=’-m64 -fast -xO3 -xarch=native64 -xchip=native -xcache=native’
$ export CXXFLAGS=’-m64 -fast -xO3 -xarch=native64 -xchip=native -xcache=native -library=stlport4′
$ ./configure –prefix=/home/felix
$ gmake
$ gmake check
$ gmake install
$ gmake distclean

i didn’t add the –enable-cxx switch for configure, since this didn’t work and i didn’t need it. note that i chose the optimization level -xO3 instead of -xO4 or -xO5 since otherwise some of the checks failed. you can try a higher level, but i urge you to run gmake check and reduce the level when checks fail.

atlas.

to build atlas, i proceeded as follows. you can replace mybuilddir with any other sensible name; that directory will contain all build specific files for that machine. note that atlas does some profiling to determine which methods are fastest, so it is better to not have anything else running on the machine while building atlas. i didn’t build the fortran parts of the library (by –nof77), as well as the fortran tests, since i couldn’t get them to link correctly. (one probably has to set FFLAGS or however the corresponding variable is called…)

$ mkdir mybuilddir
$ cd mybuilddir
$ export CC=cc
$ export CFLAGS=’-m64 -fast -xarch=native64 -xchip=native -xcache=native’
$ ../configure –nof77 –prefix=/home/felix –cc=cc –cflags=’-m64 -fast -xarch=native64 -xchip=native -xcache=native’
$ gmake
$ gmake check
$ gmake ptcheck
$ gmake time
$ gmake install
$ cd ..

iml.

building iml is rather easy. it needs both gmp and atlas.

$ export CC=cc
$ export CFLAGS=’-m64 -fast -xarch=native64 -xchip=native -xcache=native’
$ ./configure –prefix=/home/felix –with-gmp-include=/home/felix/include –with-atlas-include=/home/felix/include –with-gmp-lib=/home/felix/lib –with-atlas-lib=/home/felix/lib
$ gmake
$ gmake check
$ gmake install

ntl.

buliding ntl is a bit more complicated. it requires that gmp is already built. the whole process is more complicated since on our machine, a little tool called MakeDesc called at the beginning of the build process hangs. the problem lies in src/MakeDesc.c, when the main program calls DoublePrecision1(one) in order to find out the (internal) precision of double registers. if i replace the line

dp1 = DoublePrecision1(one);

by

dp1 = dp;

the whole process works perfectly – though maybe some things will not be 100% correct in the end. (but i’m willing to take that risk.)

$ cd src
$ export CC=cc
$ export CXX=CC
$ export CFLAGS=’-m64 -fast -xarch=native64 -xchip=native -xcache=native’
$ export CXXFLAGS=’-m64 -fast -xarch=native64 -xchip=native -xcache=native -library=stlport4′
$ export LDFLAGS=’-R/home/felix/lib -library=stlport4′
$ ./configure PREFIX=/home/felix CC=cc CXX=CC CFLAGS=’-m64 -fast -xarch=native64 -xchip=native -xcache=native’ CXXFLAGS=’-m64 -fast -xarch=native64 -xchip=native -xcache=native -library=stlport4′ LDFLAGS=’-R:/home/felix/lib’ NTL_GMP_LIP=on GMP_PREFIX=/home/felix
$ gmake
$ gmake check
$ gmake install
$ cd ..

boost.

finally, i had to compile boost. after a lot of trying and fiddling, i found out that these calls seem to work:

$ ./bootstrap.sh –prefix=/home/felix –show-libraries –with-toolset=sun –with-libraries=iostreams
$ ./bjam –prefix=/home/felix toolset=sun –with-iostreams threading=multi address-model=64 link=static install

note that i only build the iostreams library of boost. remove the –with-libraries=iostreams to (try to) build all libraries.

conclusion.

yes, the whole process is pretty much a pain in the ass. just installing the packages with apt-get on some debian-based linux, or compiling them from scratch on a gcc/g++ based linux, is just sooo much easier. but then, if you have a solaris machine standing around, why not use it to crunch some numbers for you? :-) (especially since currently, i essentially have the machine for myself.)
to compile my code, i use

$ CC -I/home/felix/include -m64 -fast -xarch=native64 -xchip=native -xcache=native -c -library=stlport4 object files and so on;

to link, i do

$ CC -m64 -fast -xarch=native64 -xchip=native -xcache=native -L/home/felix/lib -R/home/felix/lib -library=stlport4 -lntl -liml -lcblas -latlas -lgmp -lm -lboost_iostreams -lz object files and so on.

for my eighth project 52 photo i took apart a hard disk and made some macro shots. of course, especially since these are my first macro shots, this resulted in a few more shots. in case you’re interested, here are some more:

thanks a lot to the ones who provided me with the (already dead, before opening it up) hard disk! i hope they’ll like the pictures as well…