skip to main content.

posts for january 2012.

when adding a thread-specific allocator to a program of mine, to avoid terrible performance loss while using gmp and/or mpfr to do arbitrary precision integer respectively floating point arithmetic, i stumbled about a problem which seems to be fixed with newer solaris versions. in case anyone experiences a similar problem and cannot just update to a new enough solaris version, here’s some information on a dirty’n'quick fix for the problem.
more precisely, i wanted to combine boost::thread_specific_pointer (a portable implementation of thread specific storage, with dlmalloc, to obtain an allocator which won’t block when used from different threads at once. if you use arbitrary precision arithmetic on a machine with many cores/cpus (say, 30 to 60), having a single blocking (via a mutex) allocator totally kills performance. for example, on our ultrasparc/solaris machine, running 29 threads (on 30 cpus) in parallel, only 20% of the system’s ressources were used effectively. if the machine would have only had 6 cpus, the program would have run at the same speed. quite a waste, isn’t it?
anyway, combining thread local storage and a memory allocator solves this problem. in theory, at least. when i put the two things together, and ran my program with 30 threads, stlil only 60% of the 30 cpus processing power was used – the other 40% of the cycles were still spend waiting. (solaris has some excellent profiling tools on board. that’s why i like to use our slow old outdated solaris machine to profile, instead of our blazing fast newer big linux machine. in case anyone cares.) interestingly, on our linux machine, with 64 threads (running on 64 cores), the problem wasn’t there: 100% of the cycles went into computing, and essentially none into waiting.
inspecting the problem closer with the sun studio analyzer, it turns out that the 40% waiting cycles are caused by pthread_once, which is called by the internal boost method boost::detail::find_tss_data. that method is called every time a boost::thread_specific_pointer<> is dereferenced. which in my program happens every time when the thread local allocator is fired up to allocate, reallocate or free a piece of memory. (more precisely, boost::detail::find_tss_data calls boost::detail::get_current_thread_data, which uses boost::call_once, which in turn uses pthread_once in the pthread implementation of boost::thread, which is the implementation used on unixoid systems, such as solaris and linux.)
in theory, pthread_once uses a double-checked locking mechanism to make sure that the function specified is ran exactly once during the execution of the wohle program. while searching online, i found the source of the pthread implementation of a newer opensolaris from 2008 here; it uses a double-checked locking with a memory barrier, which should (at least in theory) turn it into a working solution (multi-threaded programming is far from being simple, both the compiler and the cpu can screw up your code by rearranging instructions in a deadly way).
anyway, it seems that the pthread_once implementation on the soliaris installation on the machine i’m using just locks a mutex every time it is called. when you massively call the function from 30 threads at once, all running perfectly parallel on a machine with enough cpus, this gives a natural bottle-neck. to make sure it is pthread_once which causes the problem, i wrote the following test program:

 1#include <pthread.h>
 2#include <iostream>
 3
 4static pthread_once_t onceControl = PTHREAD_ONCE_INIT;
 5static int nocalls = 0;
 6
 7extern "C" void onceRoutine(void)
 8{
 9    std::cout << "onceRoutine()\n";
10    nocalls++;
11}
12
13extern "C" void * thethread(void * x)
14{
15    for (unsigned i = 0; i < 10000000; ++i)
16        pthread_once(&onceControl, onceRoutine);
17    return NULL;
18}
19
20int main()
21{
22    const int nothreads = 30;
23    pthread_t threads[nothreads];
24
25    for (int i=0; i < nothreads; ++i)
26        pthread_create(&threads[i], NULL, thethread, NULL);
27
28    for (int i=0; i < nothreads; ++i)
29    {
30        void * status;
31        pthread_join(threads[i], &status);
32    }
33
34    if (nocalls != 1)
35        std::cout << "pthread_once() screwed up totally!\n";
36    else
37        std::cout << "pthread_once() seems to be doing what it promises\n";
38    return 0;
39}

i compiled the program with CC -m64 -fast -xarch=native64 -xchip=native -xcache=native -mt -lpthread oncetest.cpp -o oncetest and ran it with time. the result:
1real    16m9.541s
2user    201m1.476s
3sys     0m18.499s

compiling the same program under linux and running it there (with enough cores in the machine) yielded
1real    0m0.243s
2user    0m1.640s
3sys     0m0.060s

quite a difference, isn’t it? the solaris machine is slower, so a few seconds total time would be ok, but 16 minutes?! inspecting the running program on solaris with prstat -Lmp <pid> shows the amount of waiting involved…
to solve this problem, at least for me, with this old solaris verison running, i took the code of pthread_once from the above link – namely the includes
1#include <atomic.h>
2#include <thread.h>
3#include <errno.h>

copied the lines 38 to 46 from the link, and the lines 157 to 179 from the link into boost_directory/libs/thread/src/pthread/once.cpp, renamed pthread_once to my_pthread_once in the code i copied and in the boost source file i added the lines to, and re-compiled boost. then, i re-ran my program, and suddenly, there was no more waiting (at least, not for mutexes :-) ). and the oncetest from above, rewritten using boost::once_call, yielded:
1real    0m0.928s
2user    0m20.181s
3sys     0m0.036s

perfect!

yesterday i made a trip to arosa, taking the cable car up to weisshorn and walking down around 750 m (total elevation difference); it was essentially this hike, except that i stopped at prätschli as the sun began to hide behind the mountains and as i missed the turn somewhen before which lead to maran. anyway, it was a great day for a hike, with clear sky, a bright sun (i had to use my sunglasses; something i haven’t needed in lappland ;-) ), and lots of snow.
several parts of the trail were pretty steep, for example the very beginning, which was a bit scary at first. at these moments, i really wished i had some kind of stick, and was somewhat afraid to slip and slide down. but nothing went wrong, i got down safely and without slipping. i had lunch at the sattelhütte, and did a lot of small stops to take photos. well, the whole thing took me much longer than the anticipated 2 1/2 hours, but who cares. :-)

the way back was a little more adventurous. the idea was to take the regional train from arosa to chur, and then changing into an intercity to basel, jumping out in zurich. i was sleepy (the night wasn’t very long, getting up at 5 and going to bed at 1 isn’t a good combination) and dozed off several time. at one point i noticed we were still at the same point as before i dozed off the last time, but then dozed off again. eventually i realized we were sitting in the darkness, the only light coming from the other train standing next to ours. i started to get awake a bit, and to listen to the other people in the coach, to find out what’s going on. well, nobody really knew, until the conductor walked by outside and someone in my coach opened up a window and asked him. apparently, the locomotive had a defect. eventually they got light working again, though no heating, and we even started rolling a few meters, probably to enable the other train to continue, as the end of our train was apparently blocking its way. after some more waiting, almost one hour after we stopped there in the first place, a second train came up from chur and we changed into that one, which after little more waiting took us down to chur. with somewhat more than one hour of delay, and our intercity to zurich having just left the train station, i ended up in an interregio train to basel via zurich, which took like half an hour longer than the intercity. so, with around 1 1/2 hours of delay, i finally got home. at least, we had some fun in the darkness in that coach, and also later on with light and on the new train; it’s great to sit among nice people :-)

(and no, i didn’t write that text into the snow. and there was more text, like hearts and names and stuff. didn’t took pictures of these, though.)

flying southward. seeing more of the sun than during the last two weeks. such beauty.

posted in: photos traveling
tags:
places:

one reason why i like the north so much are northern lights: aurora borealis. lappland is a good place to see them, as is northern canada. thanks to the cloud cover, i missed several great auroras (judging from shots people did in this area during these days), but i still saw something. the first set of aurorae appeared when i was outside of guesthouse husky, during one night:

while visiting inari with some other guests staying at the guesthouse, we met andy keen, an aurora hunter and photographer. he gave us a lift back to ivalo, and told us a bit about his history and his fascination on auroras. well, two days later, one other guest and me went on an aurora hunting trip with andy. unfortunatley, we weren’t lucky enough to get a full-blown aurora. but we still saw something, at least! for that, andy drove us somewhere north of inari, near to a lake, where we had a great view over the lake in the direction of norway. the moon was shining bright, and without the stars it almost seemed like daylight. (i guess without the moon the aurorae would have looked more intense as well…) here are some impressions from the lake:

we later continued to another very nice spot, of which i already showed two photos here, and on the way back we stopped on a bridge where we got another wonderful view. (the last four photos of the previous post show these locations.) unfortunatley, both times without auroras. but that’s the problem of aurora hunting, you’re depending a lot on luck. (according to murphy’s law, there probably was a wonderful intense aurora as soon as we were in bed ;-) )
if you’re ever around this area and want to see auroras, do not have a car (to escape the clouds) or just don’t know what to look for, ask andy. he’s a nice guy :-) and it’s nice to be able to start dozing off while someone drives you home in the middle of the night…

the following photos were taking during night. some of them look like taken at daylight, wouldn’t there be stars at the sky… especially with full moon, the snowscape is just so bright.
the photos in the second row show the guesthouse husky, and the left photo in the third row is from näverniemi lomakylä. the last two rows are somewhere north of inari, the other rows are near to the guesthouse and/or the lomakylä.

the following photos show the winterish lappland in its full beauty. the first two photos show part of the näverniemi lomakylä.
essentially all photos have been taken on the second and the eighth of january.

how is lappland in winter? how is polar night? something i had to find out, and so i was here for almost two weeks, staying in ivalo, a village in northern lappland which is now somewhat familiar to me.
the first three days i was staying at näverniemi lomakylä, and the rest of the time at guesthouse husky. two wonderful places!
i was somewhat lucky and unlucky. lucky in the sense that usually, it is much colder here, around -15 to -25 degrees; i mostly had -5 to -15. not that -25 scares me, having lived for two years in calgary. but doing a nice walk at -15 is just much nicer than at -25… :-) i was unlucky in the sense that most of the time, it was overcast. only very few days lacked the dense cloud cover which is painting the snowscape in grey. still, a beautiful grey! the first set of photos i want to present you is from the overcast daytime, and mostly looks like black and white, or blue and white. the truth is somewhere inbetween, as the light here is quite bluish.
i also did a husky tour on one of the greyish days, as you can see below.

(you can get the large version by clicking here directly.)

posted in: panoramas
tags:
places:
posted in: photos
tags:
places: