Verktyg - or how to tame signals and slots

Using the meta-object functionality of Qt gives every C++ developer a mighty tool at hand that makes many daily tasks solvable in an easy and elegant way. You can decouple components from each other via signals and slots, communicate over thread border in a safe way, add custom properties to any QObject based object and invoke methods via QMetaObject::invokeMethod() delayed by the event loop. However all these neat features come at a certain price, we basically lost type checking by the C++ compiler. This friendly companion tells us whenever we use the wrong type for a variable, pass the wrong number of arguments to a method or try to assign variables of incompatible type. When it comes to the QObject::connect() or QObject::disconnect() methods, our beloved C++ compiler reaches its limits, though. The following statement

QObject::connect( timer, SIGNAL( timeout() ), this, SLOT( slotTimeout() ) );

is processed by the preprocessor and converted to

QObject::connect( timer, "timeout()", this, "slotTimeout()" );

So while the meta object compiler (moc) can parse the first representation and extracts all information it needs, the C++ compiler only gets two 'const char*' parameters where it doesn't know how to interpret them further. But who will make sure that the object 'timer' really provides a signal with the signature 'timeout()' and the 'this' object provides a slot with signature 'slotTimeout()'? Currently these information are only checked by QMetaObject on runtime and if an error is detected, a message similar to the following is printed to the standard output:

QObject::connect: No such signal MyClass::foobar()

The problem with runtime checking is, that errors can only be detected if the code is really executed during testing. However today's software is so complex that it is mostly impossible to execute all possible code paths during the test phase, we have to find another way...

So when do broken signal/slot connections occur? A quick search over the KDE code base shows that broken connections exists mostly in outdated or seldom run code (e.g. unit tests !!!) and in code that is under heavy development and refactoring. During refactoring signals and slots are moved around, will be renamed or removed completely from the code base. While all direct calls to the missing/renamed methods will trigger a compile error, the outdated signatures inside the QObject::[dis]connect() calls won't be noticed and lead to difficult to find bugs (yes, I mean the really ugly ones...).

After 8 years of studying computer science (Rome wasn't built in a day either ;)) it is now the time to write my diploma thesis, and since until now there is no application available that solves the problem described above, I asked KDAB Germany whether they would be interested to act as supervisor for a thesis that will address this issue. KDAB does a lot of software porting from Motif, MFC and Qt3 towards Qt4 and faces the problems of broken signal/slot connections in a larger scale, so they agreed and now Till Adam (aka KDE-PIM/KMail dude) is my official supervisor :)

Since the work started already on April 1th (no joke ;)), we have a working application now. Its name is 'Verktyg' and it is based on the KDevPlatform and KDevelop C++ parser. At this point I have to say a big Thank You! to my colleagues at KDAB Milian Wolff and Bertjan Broeksema who helped me to get into the KDevelop code quite fast and motivated me to keep on using this framework :) So what can Verktyg do for you? Verktyg can analyze CMake/Qt based projects statically for broken signal/slot connections, wrong usage of QMetaObject::invokeMethod() and missuses of QObject::setProperty()/QObject::property(). The code is available under GPL and can be cloned from http://gitorious.org/kdevcpptools/verktyg. Everything you need to know to compile and setup Verktyg is described in the README file inside the source directory. I know that deployment is still a bit tricky, you need a current version (aka git master) of KDevPlatform and KDevelop and additionally the kdevcpptools. But once you have Verktyg running, it will reward you with nice error reports like the following from kdepimlibs:

WARN: qgpgme/eventloopinteractor_win.cpp:[35:4] Class QObject has no slot 'slotReadActivity(int)' but its subclass EventLoopInteractor does, is it really used here?
WARN: qgpgme/eventloopinteractor_win.cpp:[41:4] Class QObject has no slot 'slotWriteActivity(int)' but its subclass EventLoopInteractor does, is it really used here?
WARN: akonadi/job.cpp:[280:13] Class KJob has no signal 'aboutToStart(Akonadi::Job*)' but its subclass Job does, is it really used here?
WARN: akonadi/session.cpp:[135:22] Class QIODevice has no signal 'disconnected()' but its subclass QAbstractSocket does, is it really used here?
WARN: akonadi/contact/contactdefaultactions.cpp:[51:13] Class QObject has no signal 'urlClicked(KUrl)' but its subclass ContactViewer does, is it really used here?
WARN: kontactinterface/core.cpp:[92:4] Class QObject has no slot 'slotPartDestroyed(QObject*)' but its subclass Core does, is it really used here?
ERRO: kblog/tests/testlivejournal.cpp:[358:11] Class LiveJournal has no signal 'fetchedUserInfo(QMap)'
...

As you can see there are some warnings, where Verktyg cannot decide whether the signal or slot is valid or not, because an implicit type (QObject) is passed to the connect statement instead of the explicit type, but at least it gives you some hints which makes the manual review a lot easier (Thanks to Volker Krause for the idea!). But there is also an error, which in fact is a real error that should be fixed... (did I mention outdated unit tests already?!? ;)).

So if you like to play around with Verktyg and need some help to set it up or if you have some nice ideas how to improve it, feel free to contact me via mail or ping me on IRC.

Happy analyzing!

CalDAV/CardDAV/GroupDAV Support for Akonadi

Another month, another new resource for Akonadi ;) Well, actually the resource has been started by Grégory Oestreicher 3 month ago and just recently I came across the code in playground and gave it a try. After some code cleanup (to simplify the later move to kdepim/runtime/resources) I started to refactor some parts of the resource to better match the design of Akonadi. Some of the features have been removed during that work (e.g. only reload data that have been changed on the server), but they will come back in the future.
Initially the resource was designed to support the CalDAV protocol and the calendaring part of the GroupDAV protocol, because these two protocols are basically the same, with some small differences, most of the code for listing, loading and saving events and todos can be shared.
However the GroupDAV protocol supports handling of contacts as well and there is a new standard, called CardDAV, which is the contact handling equivalent to CalDAV. The logical consequence was to add support for CardDAV and adapt the GroupDAV implementation to handle contacts as well. While we are still working on iron out some bugs and test the resource against all free available groupware servers, we plan for the future...

At the moment you'll see a configuration dialog like this:

Here you can configure which protocol to use, what the URLs for the protocol handlers are and what login credentials shall be used. Unfortunately that's much to technically. In the ideal world one would have a configuration dialog that provides a list of supported groupware servers and input fields for the login credentials, everything else should be setup automatically by the resource. And that's indeed the next big task on our TODO list, so be excited :)

Now follows another screenshot of akonadiconsole with two resources loaded (one for CardDav and one for CalDav) that a configured to access the SOGo demo system:

Open-Xchange Support for Akonadi

My colleague at credativ GmbH, Roland Wolters, has already blogged about it, KDE 4.5 will have decent support for the Open-Xchange groupware server again. Today I moved the Akonadi resource from trunk/playground/pim to trunk/kdepim/runtime/resources, so it should be a bit easier to test it for the brave ones of you. KDE 3 already had a resource (based on the old KResource framework) that supported Open-Xchange, however it also contained code for accessing the SLOX (SuSE Linux Open Xchange) groupware server, which is discontinued in favor of Open-Xchange. So the new resource dropped support for SLOX, which makes the code a lot cleaner and easier to understand ;) Furthermore the Akonadi framework provides most of the useful stuff like handling offline mode or providing a cache, so these features we got for free in the new resource.

So if you want to test the Open-Xchange support, at first you need a Open-Xchange server to connect to... in my case I was allowed to use the company server for testing and development, so the boring work of setting up a server was already done ;) However there are several HOWTOs available on the Internet, so if you have no working server yet, it's only a question of diligence. The next step is to add the Open-Xchange resource to Akonadi. As early tester you can start the akonadiconsole executable from console and choose the 'Open-Xchange Groupware Server' resource. Now the configuration dialog appears where you can enter the URL to the server and the login credentials:

The dialog also provides a button for testing whether the connection can be established. In my setup, the server is running on the company host and port 433 is forwarded via SSH to my local port 1443, therefor the strange URL ;)

After the configuration dialog is closed, the initial loading of the data is started and when this is done, you should see all available folders of the Open-Xchange server in the collection tree:

Every user has her own private address book, calendar and task list under 'Private Folder'. You can add additional folders there from within KAddressBook and KOrganizer. Below 'Public Folders' the folders are listed where all users have read/write permissions for. 'Shared Folder' contains sub folders which have been published by co-workers with restricted permissions. The 'System Folder' contains the global, LDAP based, address book. If you start KAddressBook now, all address book folders will be listed there:

As you can see, all contacts from the web interface show up in KAddressBook as well. In opposite to the old resource, the new one has support for contact groups, so all distribution lists from the server are accessible in KAddressBook now. Starting the Akonadi based KOrganizer from trunk will show you the following:

Both, the private calendar and task list show up and can be included into the current view. Again you can see all events from the web interface in KOrganizer as well.

So porting the old resource to Akonadi was really fun, because this time we have a stable base (read Akonadi) we can rely on and most of the difficult to implement stuff you get for free. The rough port took me around 10 hours (including a lot of code cleanup), so I want to encourage everybody to give it a try to implement new resources or port existing ones to Akonadi, because without the resources the new shiny applications are quite useless ;)

Akonadi and PostgreSQL

As you might know, Akonadi uses a relational database system for storing meta information about the PIM data it manages. Since the first official release this has been MySQL because it provided all features we needed and was shipped with every modern Linux distribution. However there have always been users complaining about the usage of MySQL. They claim it is big, slow and has not the right license.... Leaving the FUD away, it can never be wrong to support multiple database systems, so during the last months we got several contributions from the community that allowed Akonadi to use PostgreSQL as its database. Unfortunately the patches only allowed to make use of an already running and properly configured server, which is something you can't expect from a user who just wants to read emails or looking up something in the address book.

So yesterday I took a couple of hours and checked whether it is possible to implement PostgreSQL support like it is done for MySQL, namely that Akonadi starts its own instance of the database server with a custom data directory and connects to it via unix domain sockets, so every user can have its own database server running. At the end of the day I had a working solution and after crosschecking with other PIM developers that my changes won't break anything I committed the patch this morning to trunk. If you want to test it, simply rebuild the Akonadi server from trunk, cleanup your Akonadi installation and configuration and change the content of $HOME/.config/akonadi/akonadiserverrc to

[%General]
Driver=QPSQL

[QPSQL]
StartServer=true

Now after a restart PostgreSQL should be used instead of MySQL. At this point I'd like to ask you, dear reader, for some help. The code that searches for the postgres executable has some paths predefined where the executable could be located. Unfortunately that differs from distribution to distribution. So can you tell me in the comments where the executable can be found in your distribution, please? Under Debian it is /usr/lib/postgresql/8.4/bin/, where is it under SuSE or Fedora?

Thanks for your help!

KAddressBook and Akonadi in KDE 4.4

On the last annual KDEPIM meeting there has been a lot of discussion how the further porting of the PIM applications (kmail, korganizer, kaddressbook etc.) shall proceed. The Akonadi server is stable and ready for day-to-day use and the client library is powerful enough to let a PIM application do most of its tasks. Because of the short time frame and the different complexity of code base, we decided to port KAddressBook to Akonadi for KDE 4.4 and port the other PIM applications for KDE 4.5. While the work on KAddressBook takes place in SVN trunk, the other work is done in a separated branch that will be merged back to trunk after the 4.4 release.

Working on the port of KAddressBook has been real fun, because I had the chance to throw away all the ugly, historically grown code and restart from scratch. On the one hand most of the storage management is handled by Akonadi now which reduces the code size enormously, on the other hand most basic functionality, like editing or viewing a contact, has been implemented as separated components, that are part of a library and therefor can now be used by other applications as well. So the whole code base of KAddressBook is much cleaner and well structured now. But also GUI wise I had the chance to improve the stale user interface from KAddressBook 3.5. The MacOS X user might recognize the overall appearance of the new KAddressBook UI, yes, I have to admit, the new design is inspired by the MacOS X address book ;)

Now I'll give you a small tour through the new KAddressBook with some nice screenies:

When you start the application you'll see a window like that one

On the left side you have the list of available address books, by default an address book 'Personal Contacts' will exists that contains all your contacts from your old KAddressBook installation (better said the ones from $HOME/.kde/share/apps/kabc/std.vcf). In the middle you see the list of all contacts that exists in the currently selected address book and its sub folders. On the right side the detailed information about the currently selected contact is shown. With the port to Akonadi, not only contacts are listed in the middle view, but also contact groups (TCGFKADL 'TheContactGroupsFormallyKnownAsDistributionLists'), so they are first class citizens now and integrate much better than in KDE 3.5.

The toolbar has been cleaned up and the user interface is as simple as possible. If the 3-pane look is still too complex for you, you can switch to the 'Simple UI Mode', than KAddressBook will look like the following:

Here you can quick search for a contact or contact group via the search line or iterate over the contacts with the 'Previous' and 'Next' buttons. This view should be enough for everybody who just wants to look up contacts quickly and seldom changes their data.

If you really want to add a new contact or change an existing one, you'll be using the new contact editor component, which is like already said available for every other application as well, so no strange dbus calls or exec('kaddressbook --new-contact') statements anymore.

In the 'Contact' tab you can enter the basic information of a person that you need to contact.


The 'Location' tab contains the addresses and coordinates of a person.


The 'Business' tab contains information that are useful for business contacts.


The 'Personal' tab contains personal information about a person.

In the future the editor will be extendable by plugins, so other applications can put there custom tabs there as well.

Let's go back to the address book and contacts view:

A right click on the address book view opens a menu where you can add, remove and modify the single address books (which in fact are Akonadi resources) and the address book folders. Yes, you can create folders and subfolders in your address book now to organize your contacts and in a later version we will also have virtual folders that allows you to group contacts by an arbitraty search query. A right click on the contacts view opens a similar menu where you can add, remove and modify contacts and contact groups.

If you right click on the header of the contacts view, a menu will appear where you can choose which columns shall be shown.

So these are the visual changes in KAddressBook, but under the hood there has been many changes as well. By default the contacts are stored now in a directory, each contact and contact group in its own file, so editing one contact will change only this file. The directory is located under $HOME/.local/share/contacts, so we can migrate later without having to care about $HOME/.kde3 vs. $HOME/.kde4 issues. Akonadi brings further features for free, for example offline mode with groupware servers, so you can do your changes offline and when you go online later on, the changes are transferred to the groupware server.

At the end of this blog only one warning, don't expect the current KAddressBook to be as feature rich as its previous version. This has been a complete rewrite and I integrated only the features that made really sense IMHO.

Pimp my Nepomuk

From time to time people show up on the #akonadi IRC channel and complain about the dependency of Akonadi on Nepomuk, because Nepomuk is so slow and takes 100% CPU when they start their desktop. The 'bad' news are, we won't drop the dependency on Nepomuk, because this is a technology no other desktop environment provides so far and it gives us all the nice features we want to have (from a Pimsters point of view ;))

So what's about the good news you might ask? Well, for those of you who have Nepomuk running with 100% CPU usage, we can make Nepomuk a lot faster!

When you execute the following command on a shell

qdbus org.kde.NepomukStorage /nepomukstorage usedSopranoBackend

you will see either 'redland' or 'semsame2' as output.

If your Nepomuk uses Sesame2 and it is still slow, your hardware must be really old (and I mean really, really old ;)), it works fine here on a Laptop with 500MHz and 256MB RAM...
For most of you, who see a high CPU usage of Nepomuk, however the Redland backend is used. So what is this backend thingy about? Nepomuk is a KDE specific high-level API for Soprano, which itself is a Qt-only storage solution for RDF (semantic) data. But Soprano doesn't implement the actual storage, it just forwards this task to its backends where currently two stable ones exists, the C++ based Redland and the Java based Sesame2. The reason for the high CPU usage is, that the Redland backend is much, much slower than the Sesame2 backend!

The careful reader might wonder now "Why is the C++ based backend slower than the Java based?" Isn't Java always slower than C++? Not in this case, the storage of RDF data and the querying involves many clever algorithms, so the C++ backend is bit here by the complexity theory. The Sesame2 backend in opposite has implemented many optimizations on the algorithm level and beats Redland performance wise.

So as strange as it sounds, to get a faster Nepomuk you have to install Java.

Yeah, I know, many people will scream now "WTF, why do I have to install this Java crap?!?", well, actually it could be that you have already installed it, OpenOffice for example brings along a JRE and to be honest, the reputation of Java is worse than the current implementations actually are. But back to pimping up Nepomuk...

If you use the KDE packages from a distribution, check whether there exists a package soprano-backend-sesame and install this one. If there is no such package available and all the other Soprano related packages do not contain a file
$KDEDIR/lib/soprano/libsoprano_sesame2backend.so bug your distributor to create them or do it yourself and publish them ;)

For the brave guys that use KDE compiled from sources, you need the following things:

• The Java development package (under Debian sun-java6-jdk)
• The Java runtime package (under Debian sun-java6-jre)

Now make sure that your compiler can find the development files by adjusting the LD_LIBRARY_PATH to contain the Java library directory:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/jvm/java-6-sun/jre/lib/i386:/usr/lib/jvm/java-6-sun/jre/lib/i386/client/


(this are the needed paths under Debian, your paths might look a bit different).

Then just recompile Soprano from kdesupport (remember to clear the CMake cache before). CMake should output in the status report whether it has found the Java headers and libraries or not, if not, check your LD_LIBRARY_PATH variable again. If everything compiles fine, after a 'make install' you should have the file
$KDEDIR/lib/soprano/libsoprano_sesame2backend.so
available and after a restart of Nepomuk the qdbus command, I listed above, should return 'sesame2' now. And maybe you have already noticed, Nepomuk doesn't run with 100% CPU usage any more ;)

I hope this blog helps to reduce the prejudices against Nepomuk, because this software really rocks! Let's hope the distributors will finally ship sesame2 based Soprano packages!!!

W3C XML Schema validation with Qt

For all of you who want to make use of advanced XML processing with Qt and KDE, here is a preview story. Now we can maybe get rid of the libxml2 dependency at several places in KDE.