For openSUSE 11.1 the KDE Updater Applet will switch from the zypp backend to the PackageKit backend by default. Authorization is done via PolicyKit-kde:

A KPackageKit package will be available in the online repository for those who don't like the YaST Qt Package Manager.

It's been a while since I've blogged, mostly because of being fairly busy at work (more on that once the refactor branch is done), though I have found time for some work on a couple of C++/Qt hobby projects:

Jerboa's had a fairly major UI change (well, given the amount of UI there is, there's not much to change), and while the toolbar takes up much more space, I think it's better and more consistent. I also got around to making a DMG for mac users (windows users: MSI coming soon, builds and works fine though - linux users: cmake/qt4.4 from source). I've still not managed to hit my feature plan for 0.3 yet - when it's done ¬_¬.

I've also been working on a typing game, called TypoFail - only source so far for it though, and a screenshot or two:

The first is a clone of the amazing timewaster TypeRacer, though without the lag issues that annoy me over wireless; the latter is a clone of the venerable 'typespeed' game.

In the first mode, you type the quote as quickly as possible, and your score is your WPM. In TypeSpeed mode, your score is the number of characters in complete words you typed before you fail (10 words reaching the right hand edge of the field is a failure).

The game supports multiplayer (and pops up mini game windows showing your opponent's state), though requires a manually set-up standalone server at the moment. One neat thing about the quotes mode is that it reads fortune-format files, so you can easily load up whatever quotes you like - it'll even recognize ones which look like a dictation/IRC log:

Both of these applications are pure Qt4, and work fine on Windows, Linux, and OSX - though TypoFail's networking requires JsonQt, which it contains in a git submodule.

Well, maybe not right now ..

Since I doubt you’d be able to understand what I mean from the lolcat-speech title, let me try to summarise it in a language that nears a lot more what people actually speak (yes I know it’s still going to be too technical for some of the readers, but I guess that cannot really be helped).

Last night I couldn’t sleep, for a series of reason, not last that to make sure I could implement some stuff for my job while waiting for the actual definitive specs, I took three coffee cups, which while making me feel very nice, stops me from sleeping; not so nice when your neighbours woke you up two days in a row fighting, but I can manage.

Since at the time I was waiting for the chroot to complete some builds so I could check and submit a few more bugs (the count of “My Bugs” search on bugzilla now reaches 1200 bugs and has some reserve too), I decided to try something different. I already have been adding to my git repositories changes to a few libraries I contributed to in the past enough buildsystem so that --no-undefined is added, so last night I decided to go with doing some work on ALSA upstream repositories.

I already had checked out three of the repositories when 1.0.18 was added to the tree, since I had to fix an --as-needed issue and decided to just go on and submit all the patches to upstream for merge, this time I checked out alsa-lib, added --no-undefined and then started some analysis with ruby-elf tools cowstats and missingstatic, as well as removed a few compiler warnings, just to make sure I wouldn’t be distracted by faux problems.

The result should now be that the alsa-lib and alsa-plugins libraries have a few dirty pages less, and that the code is a bit more solid than before, with added static and const modifiers where needed. It wasn’t much of a work, but I forgot once again to add -s to the git commits so I had to rewrite history to get the Signed-off-by header to all the commits; if somebody knows how to set git per-repository to always use -s when committing, I’d be very glad.

On the other hand, this task shown me that cowstats still had and has some problems, in particular, it lacked a way to separate .data.rel from .data.rel.ro sections data. This is important to distinguish between the two since .data.rel.ro is fully prelinkable, which means after a prelink it would always be loaded from the disk without further relocation, while .data could still cause copy on write because it can be changed at runtime even after relocation.

This is even further understood by noticing that shared objects built with GCC under Linux have .data, .data and .data.rel.ro, but no .data.rel which is instead merged back into .data itself. But because of this the “real” data count in cowstats is entirely out of reality. I’ll have to rewrite that part most likely.

Anyway, I’ve done my best and hopefully tomorrow more of my patches will be merged in, so that alsa-lib’s dirty pages get reduced again. Unfortunately even after my changes, with all the plugins enabled, and in the worst case scenario, libasound.so will go on requiring more than 28KiB of dirty pages per process (minus forks and various preloads). Which is not nice at all. Prelinking can reduce the dirty pages removing these 28KiB (which are all of .data.rel.ro), and then it would just require a couple of pages.

There is one question though that now is driving me nuts: hasn’t Nokia worked on ALSA for their N800 tablets? I know alsa-plugins has a Maemo plugin (which I also cleaned up a bit last night, as it had quite a few hacks on the autotools side, and an unwarranted use of pow() instead of using left shift), but I’d expect Nokia to know better about having so many dirty pages…

Anyway, for all the remaining users, I strongly suggest you look into removing some of the plugins that ship with ALSA, like the iec958 plugin if you don’t use digital pass-through. By cutting down the amount of built-in plugins you should be able to reduce sensibly the memory that alsa and the applications using alsa would be using on your system.

I also wonder why didn’t I add an USE flag to disable the alisp feature Sorry, of course I wouldn’t be able to find an alisp USE flag if I check the output of emerge -pv alsa-utils. D’oh!. Why does ALSA need a LISP interpreter anyway?

java -classpath antlr.jar org.antlr.Tool Calc*.g # generate Java source from grammars
javac -classpath antlr.jar Calc*.java Test.java # compile everything

[davyd@frobisher Calc]$ java -classpath antlr.jar:. Test "2 + 2 + 4 * 2"
ans = 12.0

Recently I've often been amazed by the ingenuity and the lengths that some people seem to want to go to, in order to be rude about KDE4. One example was a guy on Aaron's blog about the new system tray who claimed that Plasma had 'too many casts' especially dynamic_casts. 'Hey what? Huh?' I thought, as it was a bit off the wall.

However, we have all got the Plasma sources right there in the kdelibs trunk module, and so this morning I thought I'd do a quick survey to see if there was any truth in Ramsees' claims.

Firstly, how many dynamic_casts are there in the Plasma libs code?

mardigras rdale 57% cd kdelibs/plasma
mardigras rdale 58% find . -name "*.cpp" -exec grep dynamic_cast {} ; | wc -l
21

Which sources?

mardigras rdale 59% find . -name "*.cpp" -exec grep -l dynamic_cast {} ;
./applet.cpp
./popupapplet.cpp
./extenderitem.cpp
./extender.cpp
./corona.cpp
./tooltipmanager.cpp
./containment.cpp
./private/toolbox.cpp
./animator.cpp

So we have 21 casts in 9 different C++ source files. Aaron's blog currently has 111 comments, and so there are five times as many comments as there are casts. In fact, we probably have almost as many comments about casts as there were casts.

I counted many lines are in the Plasma .cpp and .h files including white space and comments. As there are about 40000, it means that we have one dynamic_cast for every 2000 lines of code. If you are familiar with the QGraphicsView api that Plasma uses as a canvas to draw on, you will know that it has quite a complicated class structure with multiple inheritance, and quite complicated interactions with the standard QObject/QWidget classes. I would say that it is pretty much impossible to use it without doing any casts and that we can say with quite a lot of certainty that 21 casts in Plasma is a very reasonable figure.

I counted the static_casts and found 101, which again is pretty much what you would expect given a C++ library of this size and complexity.

OK, there you have it then - I am happy to confirm that the Plasma code is well written and well designed from the point of view of cast usage.

In my previous post I’ve noted that there are some cases where --as-needed stops a program from building even though it’s not because of an indirect link. I like to call this class of failures the “misguided link” failures.

Consider the following diagram showing such a case:

We have a given software, linking to libssl and instead using libcrypto. This is the inverse of the indirect case I wrote about last time, but it still features a link relationship with no use relationship, which is going to be cut by --as-needed. This is one of the most interesting cases since it’s really difficult to identify without going to check either the source code or the missing symbols. It’s not limited to OpenSSL libraries, it’s actually pretty common in general, but it happens quite a lot with them since people forget that OpenSSL is more than just libssl.

So how can we identify this problem? Well the first issue here is to identify what can cause this. Let’s say we have a simple software that calculates the MD5 of its standard input, something like this:

#include <stdint.h>
#include <stdio.h>
#include <openssl/md5.h>

int main() {
  MD5_CTX md5;
  uint8_t md5digest[MD5_DIGEST_LENGTH];
  int i;

  MD5_Init(&md5);

  while(!feof(stdin)) {
    char buff[4096] = { 0, };
    size_t read = fread(buff, 1, sizeof(buff), stdin);

    MD5_Update(&md5, buff, read);
  }

  MD5_Final(&md5digest[0], &md5);

  for(i = 0; i < sizeof(md5digest); i++)
    printf("%02x", md5digest[i]);

  printf("n");
  return 0;
}

Now if we try to compile this on a system without forced --as-needed (and no --as-needed in LDFLAGS) linking it with -lssl, it will work just fine

% GCC_SPECS="" gcc md5-ssl.c -o md5-ssl -lssl
% scanelf -n md5-ssl 
 TYPE   NEEDED FILE 
ET_EXEC libssl.so.0.9.8,libc.so.6,libcrypto.so.0.9.8 md5-ssl 
% ldd md5-ssl 
    linux-vdso.so.1 =>  (0x00007fff11bfe000)
    libssl.so.0.9.8 => /usr/lib/libssl.so.0.9.8 (0x00007f070961e000)
    libc.so.6 => /lib/libc.so.6 (0x00007f07092ab000)
    libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007f0708f19000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0709870000)
    libdl.so.2 => /lib/libdl.so.2 (0x00007f0708d15000)

but if we try to compile it with forced --as-needed, or even just --as-needed in LDFLAGS, the results are quite different:

% gcc md5-ssl.c -o md5-ssl -lssl   
/tmp/.private/flame/cc8kRKqi.o: In function `main':
md5-ssl.c:(.text+0x10): undefined reference to `MD5_Init'
md5-ssl.c:(.text+0x8d): undefined reference to `MD5_Update'
md5-ssl.c:(.text+0xae): undefined reference to `MD5_Final'
collect2: ld returned 1 exit status
% GCC_SPECS="" gcc -Wl,--as-needed md5-ssl.c -o md5-ssl -lssl 
/tmp/.private/flame/ccVWCirl.o: In function `main':
md5-ssl.c:(.text+0x10): undefined reference to `MD5_Init'
md5-ssl.c:(.text+0x8d): undefined reference to `MD5_Update'
md5-ssl.c:(.text+0xae): undefined reference to `MD5_Final'
collect2: ld returned 1 exit status

A lot of people at this point would be thrown off since the library is there, after the source files (or object files), there are no commodity libraries involved, so the linking line should be correct. But instead it fails, and the problem lies in using the wrong library.

As the name tells you, libssl contains functions that are used for implementing Secure Socket Layer, while MD5 is also used for the implementation, it’s not part of the interface. And indeed, MD5 functions are not part of the library’s interface.

Now, since even the man page for these function does not tell you which library to find them in (while most Linux, *BSD and Solaris man pages tell you which library a function comes from), you have to rely on either experience, or test to find which is the correct library.

Let’s try two different approaches here, just so that people can understand how I end up debugging these things in the first place.

To begin with, let’s check whether libssl provides the symbols we’re missing, we don’t expect it to since the link failed; easy way to do this? nm and grep:

% nm -D /usr/lib/libssl.so | egrep 'MD5_(Init|Update|Final)'
%

There is no defined nor undefined symbol with those names, which means there is no MD5 interface defined nor used in that library. Which explains why the link failed. Now since we know the build works without --as-needed we check which library libssl brings in as dependencies:

% ldd /usr/lib/libssl.so
    linux-vdso.so.1 =>  (0x00007fff1dbfe000)
    libcrypto.so.0.9.8 => /usr/lib/libcrypto.so.0.9.8 (0x00007f2d1551d000)
    libc.so.6 => /lib/libc.so.6 (0x00007f2d151aa000)
    libdl.so.2 => /lib/libdl.so.2 (0x00007f2d14fa5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f2d15b28000)

The first library is the virtual dynamic shared object of the Linux kernel, let’s ignore it; the last is the dynamic linker (or loader) itself, which we also want to ignore; we can exclude libc or otherwise the program wouldn’t have failed, since that’s always brought in. We’re left with two candidates: libdl and libcrypto. Now let’s be very dumb and ignore the name “crypto”, as well as ignoring that libdl is home of dlopen() and other known functions, and look in the two of them for the symbols:

% nm -D --defined-only /lib/libdl.so.2 | egrep 'MD5_(Init|Update|Final)'
% nm -D --defined-only /usr/lib/libcrypto.so.0.9.8 | egrep 'MD5_(Init|Update|Final)'
000000000006a5e0 T MD5_Final
000000000006a5a0 T MD5_Init
000000000006a6e0 T MD5_Update

So we found the problem, and indeed you can try yourself that requesting -lcrypto directly in the build of the program above will make it work just fine with and without --as-needed, with the added benefit that libssl is not being loaded when running the software.

Now this is a slightly boring and long approach, the alternative approach, which work just fine in Gentoo, requires just one command:

% scanelf -ql -s +MD5_Init
MD5_Init  /usr/lib64/libcrypto.so.0.9.8
MD5_Init  /usr/lib64/libgnutls-openssl.so.26.11.3

The scanelf call we have here will go searching for the correct library we need, although it might confuse you since it might report different implementation or totally unrelated libraries in case of symbol collisions (which is something I use to identify broken software by the way). Note that here I just targeted one symbol, the reason for this is that the current version of scanelf from 0.1.18 is not working properly with regex-based search; in the current CVS version you could be using scanelf -gqls 'MD5_(Init|Update|Final)', but it would just find the first anyway.

Is this easy enough to fix, in your opinion? Also consider that if software were to use pkg-config, right now, it would be listing -lssl -lcrypto -ldl, which would stop --as-needed from breaking, but is most likely going to break in the future if libssl.pc is updated to use Require.private to list libcrypto.

A few weeks ago I got a smart card to use with gpg for hardware encryption. I'm no security fanatic but I like the idea, so I bought a "lots of different cards all in one" reader. I got a MSI StarReader SMART which should support smart cards and was available locally (strange habit, I like to go to real shops instead of the online competition sometimes).
I played around with it, but it seemed to just sit there and do nothing (except read every variant of useless memory card). What made my day is that after only one mail to Ludovic Rousseau with some info about the device and getting a response the same evening, it started working. After adding its usb id it's listed on the ccid driver page
A big thank you to Ludovic Rousseau!
Time to get it to work with gpg and mail now.

KDE 4.2 is in bug fixing mode and so is Marble. Time to have a look at things that got implemented right in time for Marble 0.7: Henry de Valence has been one of the most active Marble core developers during the last few months: He has implemented several exciting Marble features already.

For Marble 0.7 (KDE 4.2) he implemented MarbleRunners. This is a pretty nice feature, which is still a bit hidden, but that is probably going to change soon:

1. Make sure you have internet -- Oh, ok, yes, I just wanted to make sure ...
2. Start Marble (e.g. from KDE 4.2 Beta1 or trunk)
3. Find the "Search" field and type in: Playmobil
4. Wait for the results to arrive!

What happens is that on pressing the Enter key Marble will send the search query to several "service" threads in the background called "MarbleRunners". These threads will return the result as soon as it is available. In this case the OpenStreetMap-Runner has returned four matching results from the OpenStreetMap-Server!

Ok, let's try another one: Imagine, you are somewhere in Finland on holidays ("Tervetuloa!"), it snows and you feel really hungry. In this situation Marble can be a true life-saver ("Hyvää ruokahalua!"):

The source code for the OpenStreetMap-Runner is based on source code by Jens-Michael Hoffmann who also created the OpenStreetMap integration for Marble. Henry de Valence has taken it and created a Marble runner out of it. Additionally Henry has created a Coordinate-Runner:

Enter e.g.

or just:

and press the Enter key. The latter will almost immediately return a result from the GeoCoordinate-Runner and several results from the OpenStreetMap-Runner trailing in a few seconds afterwards:

The best thing is however that Henry has made a HOWTO for creating MarbleRunners. So now you can create your own MarbleRunner yourself! The example shows how easy it is! And the description ensures that you can do this even if you are a beginner.

Just go here to get the HOWTO and just look here to get instructions on how to compile Marble.

I could imagine lots of runners: What e.g. about a Wikipedia-Runner (we don't have that one yet, ...)? What ideas do you have? Please let us know or even better: send us your patch! We are reachable via marble-devel@kde.org. Or join us on IRC ( #kde-edu, Freenode: irc.kde.org )! Of course we also appreciate all kinds of patches that improve the current MarbleRunners as well!

You've seen the code. Now watch the video.

Albert White has put together a blog entry today about OpenSolaris Code Swarms.

There is a great video, which is a graphical representation of the development over time of the Image Packaging System, (which I started checking packaging changes into a couple of months ago). I just love how large scale changes are represented (such as the integration of a new WOS build with many new or changed packages).

Image Packaging System Code Swarm from Albert White on Vimeo.

(Thanks Alan).

[Technorati Tag: IPS]

[Technorati Tag: OpenSolaris]

> Read More... | Digg This!

Join an OSUG here. Propose a new OSUG here.

I hope the images appear correctly, please just leave me a comment if they don’t and I’ll try to fix them.

I think that after my writeup and Robert’s bugspree some people might have the wrong idea about the relationship between the --as-needed and the --no-undefined flags.

Let’s begin to say exactly what --no-undefined does: it makes the linker reject building targets that have undefined references that are not satisfied by any of the libraries it links to directly and indirectly. The linker already rejects this for final executables, but for a series of reason the default is to allow undefined references in shared objects. But if you have a library A that calls functions from the library B but does not directly link to it, with --no-undefined the linker will refuse to build A entirely; the default would be for A to still build, and the software SW that is going to use A to be forced to explicitly link in B. The following image shows what I mean in the form of graph:

In the above image you can see the “Use relationship” and the “Link relationship” not being balanced, and here comes the problem, since --as-needed has, as task, to remove the link relationships that are not paired with an use relationship. Now just to make sure everybody is on the right page, the reason why --as-needed exists is that, thanks to the original conception of libtool and pkg-config and others, we can easily have programs whose linking diagram is something like the following:

If you look carefully you can see that there are some linking branches that are not actually used at all; this is because the linker, by default, links in whatever you tell it to link, so you can easily link in a program some libraries that are never ever used, which is not only a waste of time, but it also wastes time and resources at link time since the linker may have to take care of relocations, wastes time during symbol resolution because the extra libraries need to be scanned too, and might take up resources if they have constructor functions for instance, that will cause initialisation functions to be executed, which might open datafiles, allocate data structure and so on.

Now, what we’d like would be something like the following diagram, which shows the linking reduced to the actual needed parts, still following the same rules as the original:

This would reduce the amount of objects loaded to the minimal, needed part, so that no foreign object gets to be loaded at runtime, or linked in at build time for what matters. Unfortunately, this is a dream: the way the linker works, --as-needed produces different results than what you see here, it produces this:

I’ve changed colours for the object in this diagram: the yellow objects are the ones that lack a linking relationship, and thus won’t be loaded up, the red objects are those which are broken, since they use an object they don’t link to. You can see that there actually is an exception to that rule, since the yellow object in the middle of the graph uses the blue one at the right end of it, but don’t link directly to it; while it’s probably a good idea to make sure that all you use is also linked in, that situation is legal since the link to the highest object is available indirectly from another object.

Now, --no-undefined would have caught the two broken objects at their build-time rather than when they were to be used by another project, but as I’ll try to explain in the next days, this option is not a panacea, but it helps to identify the issues up in the stream. On the other hand, there are some situation where --as-needed finds trouble that --no-undefined wouldn’t identify early on.

Since people often tell me I write way too much in a single blog entry, I’ll try to wait adding the rest of the content till this entry is digested, the next chapter hopefully in two days, in the mean time feel free to write any question you have in the comments, so I can answer, either on the comments (as you may guess I read them but not always have time to address them directly), or in the next posts.

By the way, the speed to which I can write these article depends directly on the amount of caffeine in my bloodstream, so if you wish to have more content written faster, you can always help me by getting me some good coffee beans, I have never tried java for instance.