Hi!

I arrived back home from Akademy just a day ago and I already miss it. I enjoyed every single moment of it and had lots and lots of fun. Thanks everyone for making this such an awesome event, and especially to the local team. They did an incredible job!

This blog however will not be about Akademy (I will write one maybe later), but about Akonadi, core component of our PIM suite. As you probably already read or heard, Volker Krause has handed over to me maintainership of Akonadi.  It means really lot to me and I’ll do my best to be at least as good maintainer as he was (if that’s even possible) and I would like to thank him for his outstanding job he did writing and maintaining Akonadi.

I really believe in Akonadi, I like design of the framework and admire all the work guys have done on it since the beginning, long before I even dreamed about becoming a KDE contributor. I also believe that having a powerful and well-working PIM suite is the key for success of KDE (not just) in the enterprise world. Akonadi is more powerful than most of the competition out there, we just now need to focus bit more on stability and performance. I think we are doing pretty fine with stability, so I want to focus mainly on the performance side of Akonadi. In this bit more technical blog post I want to write about what I did recently, what I’m doing know and what are (some of) my future plans with Akonadi. As usually, huge thank you to Volker for his ideas, suggestions and comments about my ideas. We had a great discussion during Akademy and we (theoretically) solved many problems and bottlenecks that were bothering Akonadi for a long time, but nobody had time to look into it.

Batch Notifications

I started working on batch notifications after the KDE PIM sprint in March, merged it in May (I think) and hopefully have resolved all regressions by now. Akonadi server uses notifications to inform all clients about changes, like new items, changed items, removed items, etc. Notification is a simple data structure that is transferred via D-Bus to all clients. Before this patch there was one notification per each item which means that marking 500 emails as read had generated 500 notifications that had to be transferred via D-Bus. With batch notifications the server can create a single notification that references all 500 items. This saves a lot of D-Bus traffic and allows faster processing on the client side. This feature will be available in KDE 4.11.

MTime-based item retrieval

This was written during Akademy and will allow Vishesh to improve start-up time of the Akonadi Nepomuk Feeder. Until now the feeder had to fetch all items from Akonadi and all items from Nepomuk on start just to compare whether everything is up-to-date. With mtime-based item retrieval the feeder can ask Akonadi to hand over only items that have been changed since some given timestamp. In most cases that means 0 items. You have to agree that fetching no or just a few items instead of all of them will give us a notable performance boost during start. Albert allowed us to backport this to 4.11, so you will get this improvement in 4.11 as well.

External payload files without path

Another thing that has been implemented during Akademy is aimed to save some of your precious disk space. Because Akonadi is a cache for your data, it means that it has to store all your emails, contacts, events etc. somewhere. Smaller records are stored directly in the database. Larger items are stored in external files on your hard drive and there is only file path stored in the database. However the path is always the same and is usually around 50 characters long, while the file name is only around 10 characters long. This patch makes sure that only the file name without path is stored in the database, saving some disk space. Clients now also have the capability to understand file names without path in server replies, so we can even reduce size of traffic  between server and clients a bit. I know that 50 bytes is not much, but multiply it by tens or hundreds of thousands of items, and it’s already worth it. As a side effect, all newly created databases will be relocatable, because all file paths are relative, not absolute. There is no plan to make an automated migration to strip the path from existing records, but I might one day implement this in the Janitor, so that users can migrate their database manually if they want to. But it’s not a priority now.

Server-side monitoring

Previously I explained that Akonadi sends notifications via D-Bus to all clients to inform them about changes. The problem here is that not all clients are usually interested in all changes. For instance KOrganizer does not care about new emails and KMail does not care about modified appointments (there are exceptions like the Nepomuk Feeder, which listens to everything for obvious reason). Yet each notification is “broadcasted” to all Akonadi clients. Each client then decides whether it’s are interested in the notification and wants to process it further or just drops it. The average number of clients is around 16, but in most cases only 3 or 4 clients are actually interested in each notification. That means that other 12 or 13 clients just drop the notification. What I’m working on right now is to move this filtering code to the Akonadi server, so that the clients can tell the server what kinds and types of notifications they want and the server will only send notifications to those clients that are really interested in it. This should save us a lot of D-Bus traffic and will fix the awkward situation when all clients are consuming CPU, even when you are just syncing one of your IMAP accounts.

Server-side change recording (IDLE)

The biggest task ahead of me. Some Akonadi clients are using feature called change recording. That means that every notification that is not dropped by the client is stored into a binary file (every client has one such file) and is removed again when the client confirms that the notification has been processed. This is used for example by the IMAP resource. When you are offline the resource is recording all notifications (about items being deleted, moved between folders, marked as read, etc) into the file and when you connect to the internet and the resource is switched to online all notifications are replayed from the file. My plan is to implement something similar to IMAP’s IDLE. Changes will be recorded on the server instead of the clients and all clients will be able to connect to the server and request all pending notifications. After that they send “IDLE” command + explanation of what kind of notifications they are interested in and the server will automatically feed them with new notifications. This is essentially continuation of the “Server-side filtering” feature, but it takes it to a completely new level. With this feature the Akonadi framework will generated almost no D-Bus traffic at all and the whole thing will be much much faster. I’m really looking forward to work on this because it’s a very challenging task and the result will definitely be worth the effort.

Volker has also submitted a few patches to reduce size of the messages sent between clients and server even more and started working on optimizing some SQL queries so that we don’t query the database for data we don’t actually use anywhere.

Of course there are more smaller ideas and improvements in the queue, but I need to keep something for next blog posts so stay tuned - there’s more coming soon! :-)

I haven’t blogged about my involvement in KDE PIM in a while, so let’s see what’s new there, especially in the Google integration part….

Reborn Google Resources

Just before the KDE PIM sprint in Berlin this month, I’ve sat down and written completely new API for LibKGAPI (the library that implements Google API and is used by the Akonadi resources for Google services). The new API is job-based, and therefore much more awesome than the old one (which is known to suck). Anyway - what does this mean? It means that the new resources are awesome as well!

Google Contacts resource now has a full support for contacts groups. All contacts are stored in the top-level collection and are linked to the respective groups, so it does not matter where you edit the contact, you are still modifying the same instance. Like in the web interface.

Google Calendar now supports limited sync, so you can choose to only sync events from last year, or last two years (the default is last 3 years) instead of the full history.

Both resources have improved status reporting, error handling, are more stable (no more mystery crashes due to unhandled exceptions thrown from LibKGAPI) and subjectively synchronization is faster too.

Murdered Google Resources

As most of you probably noticed by now, Google is planning to shut down Google Reader by July 1. It’s pitty, because I already had a fully working Akonadi resource for Google Reader ready in the akregator_port branch. Cost me lot of time and nerves. Well, the resource is not there anymore and the only memory of it is greader branch with API implementation in LibKGAPI (which will die as well sooner or later). The good news however is that I can now help Alessandro and Frank with ownCloud News and the ownCloud Akonadi resource, so that we rock when Akregator2 is out :-) I can’t wait to see what has changed in ownCloud since I installed 3.0.0 some time ago…

Upcoming Google Resources<

I have two feature requests in bugzilla: one is to support Google Bookmarks, which is fairly complicated because of missing official API and absolutely no write API. So this is not going to happen soon. The second feature request is for Google Drive KIO slave. This is much more interesting task. I already tried writing Google Docs KIO slave about three years ago and I failed epically. Retribution! There’s almost complete API implementation by Andrius in LibKGAPI git, so I plan to port it to LibkGAPI2 and see whether we can together fight the Dark side and create a nice and shiny KIO slave.

Finally, deep in the dark corners of my mind, my so far the most evil plan is slowly shaping. The plan includes modifying the current IMAP resource, reusing most of it’s code and subclassing some specific parts to build a native GMail Akonadi resource that would support some GMail-specific IMAP extensions. The main idea is to support one-mail-in-multiple-folders-at-once case. Right now the IMAP resource handles that by creating a new instance of the same email in multiple folders. My bold plan is to store all emails in Inbox and link them to respective folders. This means that marking an email as read in one folder, will automatically mark it as read in all other folders (because it’s a single instance). The IMAP resource looks scary though, so I don’t know yet when I’ll get the courage (and time) to sit down and actually start coding…I guess probably after Akademy, after I talk to some people.

Batch Operations in Akonadi

I have talked to Volker Krause during the KDE PIM sprint about how to effectively handle “Mark feed as read” in the Google Reader resource. Currently, Akonadi creates a new notification for every change, therefore marking 300 items as read generates 300 notifications, which are delivered to the Akonadi resource, which should then create 300 HTTP request to store 300 changes. You probably agree that this slightly suboptimal. (I temporarily solved the problem by caching the notifications in the resource itself and then sending a big request to Google Reader at once). The solution that Volker suggested sounds fairly simple (it’s not) - batch notifications - i.e. a single notification about single change involving multiple items. The supported changes can be flags change, deleting or linking of items. By being able to deliver single notification about mass-change to Akonadi clients and to Akonadi resources represents new possibilities for optimizations. For instance the IMAP resource could simply send a single command to add a flag to multiple emails at once, instead of doing it one by one. The same goes for other operations and other resources that are dealing regularly with operations on larger sets of items. The obvious result: performance boost! After two weeks the work is in semi-working state - it works, but it goes nuts if more than 5 items are involved. The cause is known, but solution not (but I’ll get there eventually :-) )

Akregator 2

I’m occasionally helping with Akregator2 (Akonadi port of Akregator). Recently (ok, it was two months ago… ) I’ve written Akonadi Nepomuk Feeder plugin that is feeding RSS Articles into Nepomuk and a Search window (slightly inspired by the one in KMail) in Akregator2 where you can do full-text search (+ search via other criteria, including author’s name and date of publishing) based on data indexed in Nepomuk. Obviously, when I wanted to demo that on the KDE PIM sprint I found out that it’s not working as good as I thought, so there’s still some work to be done. But in general I’m happy to say, that from time to time it finds something :-).

Ok, so that’s about what I was, am and will be working on in KDE PIM. Here I’d like to say big thank you to all KDE PIM devs, because they are doing an incredible job. Thank you!

We have done some changes to our KDE Telepathy repositories recently, that might be interesting for packagers.

We have consolidated all Plasma applets into a single repository called ktp-desktop-applets (originally ktp-contact-applet). The ktp-presence-applet repository is now empty, the master branch contains just a README, the stable branch kde-telepathy-0.5 is preserved.

We have also normalized names of the applets, so they are now called org.kde.ktp-presence, org.kde.ktp-chat, org.kde.ktp-contactlist and org.kde.ktp-contact. The ktp-desktop-applets installs an update script for Plasma Desktop that will automatically update user configuration on next login.

All declarative plugins are now located in ktp-common-internals, so that all our QML models and utils are available for developers.

These changes will be first available in 0.6 release.

Every year Red Hat Czech offers a list of topics for bachelor and diploma theses to students from local universities. The topics are usually suggested and posted by Red Hat employees who then work with the students as technical consultants and mentors.

We see this as a good opportunity for students to learn about open source software, communities and collaboration and for us to find potential future employees :-) (so there’s a great motivation too). We want students not to see their diploma or bachelor thesis as some boring obligation, but rather a “fun” way to learn new things, meet people and gain experience. And last but not least they are potential contributors to the projects they are involved in.

This year I managed to get several students I’m mentoring to pick a topic related to KDE.  Although all projects are somehow related to KDE Telepathy, it’s a good start ;-)

Today I want to introduce work of first two students, Stanislav Láznička and Artur Dębski.

Makneto++

Stanislav Láznička is doing his bachelor thesis on Brno University of Technology. His topic is “Shared Board - Makneto”, more specifically, his task is to port Makneto to Telepathy. Makneto is a whiteboard-sharing application for KDE, similar to KWhiteboard. It has been written some time ago as a diploma thesis. Since then several students have based their theses on this project (not all of them successfully though). Stanislav is now researching how to use the Telepathy framework and preparing to start the actual port soon. If you are interested in further progress, follow his blog and his git repo on bitbucket.

KTp Active

Artur Dębski (IRC: mentero), a student from Silesian University of Technology in Poland. His topic is “Touch interface for KDE Telepathy”. Yes, hang on to your hat, we are going to have a totally awesome Plasma Active application for Instant Messaging. Artur has written an extensive blogpost about his work together with some mockups. You can checkout the existing code from here, there’s already a demo app to play with!

What is your excuse for still using Kopete instead of KDE Telepathy? Oh, you can’t live without your old conversation logs? Not a problem anymore!

KDE Telepathy is now able to import your AIM, MSN, ICQ, Yahoo, Jabber or GaduGadu logs from Kopete into Telepathy logger.

When you add a new account in the Accounts KCM, we will convert the new account ID into Kopete account ID and check, whether there are any logs in Kopete folder for this account. And if there are, we ask you whether you want to import them or not.

That’s nice, right? But what about our current users, who just silently weep, longing for their old Kopete logs? Well, we thought about them, too! After starting the KDE IM Log Viewer, you will be prompted with initial logs import dialog. The dialog will appear only once. Whether you click “Import Logs” or “Cancel”, we won’t bother you with this never again. Currently I’m considering adding a main menu to the log viewer, which would contain an action to invoke this dialog again, in case you accidentally click “Cancel” for instance.

When you are removing an existing account from KDE Telepathy, you are prompted the “Are you sure you want to remove the account?” dialog. As a bonus, I have enriched it by a “Remove conversations logs” checkbox. Guess what will happen if you check it ;)

All these features will be available in KDE Telepathy 0.6.

And as usually, we are still looking for new contributors! There are some junior jobs in the bugzilla, which are a perfect start for newcomers. If you would need any help with fixing these bugs or wishes, just come talk to us on #kde-telepathy IRC channel or mailing lists.

Earn your eternal glory by fixing KTp bugs!

As some of you might have noticed, display management in KDE is not really something we could be proud of. It does not work as expected, it lacks some features and it’s not really maintained. Time to change it, don’t you think? :-)

The effort was initiated by Alex Fiestas (who is too busy to making KDE rock, so the blog post is up to me :-)). Alex has written the libkscreen library that provides information about available/connected/enabled outputs and notifications about their changes. He also intends to write a KDED daemon that would listen for these events and depending on connected monitors (every monitor can be uniquely identified by it’s EDID) it would load specific configuration. For example, docking your notebook into a docking station at work would automatically turn on a second monitor and place it left of the notebook screen (or whatever you configure the first time you do it). Undocking the notebook and connecting a data projector in a meeting room would automatically set clone mode etc. etc.

This also requires a new UI in System Settings which is the part I’m working on.

It’s written in QML and allows you to configure your displays by dragging them around rather than configuring them through combo boxes. Picture is worth a thousand words, and when it’s a moving picture, well…..

Download OGV (1.8 MB) (in real time the animations are faster and smoother of course)

The best part of all this is that users won’t be exposed to the KCM very often, because connecting an already-known monitor will configure it and place it automatically depending on the last configuration. Connecting a previously unknown output should pop up a simple window/dialog where user can quickly select whether the display should be left/right/clone of the active screen or open the KCM and perform more advanced configuration.

Right now I’m abusing the krandrtray icon for the applet. It does not provide any rich features like krandrtray though, it only has a context menu with a single action to start the KCM. This should be enough because unlike current krandr-based display configuration there most things will work automagically.

And of course we will take care of displaying these dialogs and windows on the correct screen (that is the one that is connected and enabled) :-)

Finally, we want to use KWin scripting engine to display a black overlay over the entire desktop when changing display configuration in order to hide Plasma flickering and resizing from users and make it look like a smooth transition.

Hopefully I didn’t miss anything :)

On Akademy I made a few patches for KDE Telepathy text-ui and logviewer. I continued even after Akademy, and David though it would be a good idea to make a maintainer of the KDE Telepathy LogViewer (thank you David! :) ). As a new maintainer I decided to kill all bugs, implement all feature requests and add a few of my own improvements to make the LogViewer rock.

So let’s see how the LogViewer will look like in KTp 0.5 :-)

Nicknames and Accounts

The logviewer tries to retrieve real names (or nicknames) of the contacts, so that you can easily identify the person (and you don’t have to remember who john123456789@gmail.com is). Unfortunately the names can be obtained only when you are online. When you account is disconnected, you will just see the ugly usernames. Also the list was converted into a tree, where contacts are grouped by your accounts.  You can filter the contacts either by their username of display name, too.

Plan is to have the same (similar) view as in Contact List (with fancy look and avatars).

Navigating between Conversations

Usually you only remember the contact in whose logs to read, but you don’t remember whether you should be looking to logs from yesterday, the day before or last week. Searching for valid dates in the date picker is not always comfortable. So logviewer now automatically inserts a links to previous and next  conversations to the beginning and end of each log to make jumping between them easier.

Global Search

If your memory is as bad as mine, you usually just remember a single word or term, but you have no idea who and when sent you it. Then use global search! The contacts view is filtered to contain only contacts that have at least one conversation with matching terms. Only matching conversations are displayed in the date picker and the searched term is highlighted in the message view so that you can spot it more easily.

Future?

I have some cool plans for future, some of them will make it to 0.5, some will have to wait. Here’s what you can be looking forward to:

• file transfer logging - Alin suggested that file transfers are part of conversation history and thus information about it should be stored in the logs as well., including link to the downloaded file. I agree and if I find out how to do it, I’ll add it to the logviewer :-)
• better date picker - searching for conversation dates in the date picker can be quite painful if you don’t remember the exact date or when you chat with some person only rarely, so I’d like to introduce some better UI for listing past conversations (maybe just a list of dates…)
• that’s all I can remember right now, but there will be more. I  have some ideas for the text-ui as well, so stay tuned ;-)

Do you have any ideas, suggestions, bug reports? Express your wishes in Bugzilla!

This semester I attended Advanced Topics of Linux Administration course on my university. To successfully complete the course, one must pass an exam, have a presentation and write an article on a related topic. I had a presentation about libvirt and decided to write an introduction into using libvirt’s API (the article has been published (in English) on Czech Linux portal ABCLinuxu.cz).

As an example, I have written a Plasma Applet in QML (yay, my first QML thingie!) and a Plasma DataEngine that serves data about QEMU/KVM virtual machines running under local libvirtd.

On the applet you can watch state of all virtual machines, you can boot/pause/resume/shutdown and reboot them and configure soft and hard memory limits and amount of virtual CPUs allocated to each machine.

Pictures!

I must admit that I was always a bit skeptical about QML, but my two previous encounters with QML made me really curious about it and writing this applet was really pain fun and it taught me a lot about QML.

The big advantage of the DataEngine is that it’s not constantly polling libvirt for changes, but it rather utilizes it’s events API.

Wanna try? :-) There’s a tarball here and git repo here: git://anongit.kde.org/scratch/dvratil/plasma-virt-monitor

It was fun to hack and I plan to give it some more love, possibly after Akademy.

PS: Akademy - anyone flying on Friday at 08:10 from Vienna :-) ?

This is just a short note mostly for packagers and other developers using LibKGoogle (if there are any). Due to possible violation of Google trademark, the LibKGoogle library has been renamed to LibKGAPI. Thanks to Clifford Hansen for inventing such a nice name :)

Also the license has been changed and the project is now under GPLv2+ instead of GPLv3+.

The entire API is now encapsulated in KGAPI namespace instead of KGoogle and headers were moved to %prefix%/include/libkgapi.

All these changes are now reflected in LibKGAPI 0.4.0 release, which does not contain any other improvements or fixes. I decided to go directly to 0.4 instead of continuing the 0.3 series, as I consider this a big and important change.

The Akonadi resources for Google services in kdepim-runtime will depend on LibKGAPI 0.4.0, the patch has been already submitted for review.

The git repository is still called libkgoogle, it will be renamed when it’s moved to KDE extragear (or somewhere else). Stable branch is LibKGAPI/0.4.

Tarball release: libkgapi-0.4.0.tar.bz2 (will be available on ftp.kde.org soon)

Thanks and sorry for any inconvenience.

Cheers

Hi! Nearly four weeks after the [0.3](https://dvratil.cz/2012/04/akonadi-google-0.3-arrives/ release of Akonadi resources for Google there’s a new version with just a few, but important bug fixes and improvements.

UPDATE: In order to use the Google resources, you either need KDE 4.9 (which the resources are part of), or you need to install LibKGAPI 0.4.1 (or newer) and Akonadi resources from git://anongit.kde.org/scratch/dvratil/akonadi-google-resources. See this discussion for details.

Fixed bugs and crashes:

• Bug #296541 - Uncaught exception in signal handler in Contacts resource
• Bug #297824 - Uncaught exception in signal handler in Calendar resource
• Bug #297548 - Crash at akonadi start after having added a new contact - resource
• Bug #298054 - Can’t build libkgoogle with KCal
• Bug #298518 - Unable to edit newly created events
• Bug #298519 - Deleting events incorrectly reports an error

The first two bugs were especially tricky as I couldn’t reproduce them, but many users were affected by ugly and repeating crashes. But now the “Google experience” is much much better :).

Big thanks go to Alex Fiestas who has contributed various improvements to libkgoogle to better work with 3rd party apps (so now we can be looking forward to his web-accounts wizard :) ).

Sources: akonadi-google-0.3.1.tar.gz (MD5: *fed8d9082547835ab916edd219831cf6)

Bye!

PS: I found this on Akademy wiki, so: