Hi!

I arrived back home from Akademy just a day ago and I already miss it. I enjoyed every single moment of it and had lots and lots of fun. Thanks everyone for making this such an awesome event, and especially to the local team. They did an incredible job!

This blog however will not be about Akademy (I will write one maybe later), but about Akonadi, core component of our PIM suite. As you probably already read or heard, Volker Krause has handed over to me maintainership of Akonadi.  It means really lot to me and I’ll do my best to be at least as good maintainer as he was (if that’s even possible) and I would like to thank him for his outstanding job he did writing and maintaining Akonadi.

I really believe in Akonadi, I like design of the framework and admire all the work guys have done on it since the beginning, long before I even dreamed about becoming a KDE contributor. I also believe that having a powerful and well-working PIM suite is the key for success of KDE (not just) in the enterprise world. Akonadi is more powerful than most of the competition out there, we just now need to focus bit more on stability and performance. I think we are doing pretty fine with stability, so I want to focus mainly on the performance side of Akonadi. In this bit more technical blog post I want to write about what I did recently, what I’m doing know and what are (some of) my future plans with Akonadi. As usually, huge thank you to Volker for his ideas, suggestions and comments about my ideas. We had a great discussion during Akademy and we (theoretically) solved many problems and bottlenecks that were bothering Akonadi for a long time, but nobody had time to look into it.

Batch Notifications

I started working on batch notifications after the KDE PIM sprint in March, merged it in May (I think) and hopefully have resolved all regressions by now. Akonadi server uses notifications to inform all clients about changes, like new items, changed items, removed items, etc. Notification is a simple data structure that is transferred via D-Bus to all clients. Before this patch there was one notification per each item which means that marking 500 emails as read had generated 500 notifications that had to be transferred via D-Bus. With batch notifications the server can create a single notification that references all 500 items. This saves a lot of D-Bus traffic and allows faster processing on the client side. This feature will be available in KDE 4.11.

MTime-based item retrieval

This was written during Akademy and will allow Vishesh to improve start-up time of the Akonadi Nepomuk Feeder. Until now the feeder had to fetch all items from Akonadi and all items from Nepomuk on start just to compare whether everything is up-to-date. With mtime-based item retrieval the feeder can ask Akonadi to hand over only items that have been changed since some given timestamp. In most cases that means 0 items. You have to agree that fetching no or just a few items instead of all of them will give us a notable performance boost during start. Albert allowed us to backport this to 4.11, so you will get this improvement in 4.11 as well.

External payload files without path

Another thing that has been implemented during Akademy is aimed to save some of your precious disk space. Because Akonadi is a cache for your data, it means that it has to store all your emails, contacts, events etc. somewhere. Smaller records are stored directly in the database. Larger items are stored in external files on your hard drive and there is only file path stored in the database. However the path is always the same and is usually around 50 characters long, while the file name is only around 10 characters long. This patch makes sure that only the file name without path is stored in the database, saving some disk space. Clients now also have the capability to understand file names without path in server replies, so we can even reduce size of traffic  between server and clients a bit. I know that 50 bytes is not much, but multiply it by tens or hundreds of thousands of items, and it’s already worth it. As a side effect, all newly created databases will be relocatable, because all file paths are relative, not absolute. There is no plan to make an automated migration to strip the path from existing records, but I might one day implement this in the Janitor, so that users can migrate their database manually if they want to. But it’s not a priority now.

Server-side monitoring

Previously I explained that Akonadi sends notifications via D-Bus to all clients to inform them about changes. The problem here is that not all clients are usually interested in all changes. For instance KOrganizer does not care about new emails and KMail does not care about modified appointments (there are exceptions like the Nepomuk Feeder, which listens to everything for obvious reason). Yet each notification is “broadcasted” to all Akonadi clients. Each client then decides whether it’s are interested in the notification and wants to process it further or just drops it. The average number of clients is around 16, but in most cases only 3 or 4 clients are actually interested in each notification. That means that other 12 or 13 clients just drop the notification. What I’m working on right now is to move this filtering code to the Akonadi server, so that the clients can tell the server what kinds and types of notifications they want and the server will only send notifications to those clients that are really interested in it. This should save us a lot of D-Bus traffic and will fix the awkward situation when all clients are consuming CPU, even when you are just syncing one of your IMAP accounts.

Server-side change recording (IDLE)

The biggest task ahead of me. Some Akonadi clients are using feature called change recording. That means that every notification that is not dropped by the client is stored into a binary file (every client has one such file) and is removed again when the client confirms that the notification has been processed. This is used for example by the IMAP resource. When you are offline the resource is recording all notifications (about items being deleted, moved between folders, marked as read, etc) into the file and when you connect to the internet and the resource is switched to online all notifications are replayed from the file. My plan is to implement something similar to IMAP’s IDLE. Changes will be recorded on the server instead of the clients and all clients will be able to connect to the server and request all pending notifications. After that they send “IDLE” command + explanation of what kind of notifications they are interested in and the server will automatically feed them with new notifications. This is essentially continuation of the “Server-side filtering” feature, but it takes it to a completely new level. With this feature the Akonadi framework will generated almost no D-Bus traffic at all and the whole thing will be much much faster. I’m really looking forward to work on this because it’s a very challenging task and the result will definitely be worth the effort.

Volker has also submitted a few patches to reduce size of the messages sent between clients and server even more and started working on optimizing some SQL queries so that we don’t query the database for data we don’t actually use anywhere.

Of course there are more smaller ideas and improvements in the queue, but I need to keep something for next blog posts so stay tuned - there’s more coming soon! :-)

I haven’t blogged about my involvement in KDE PIM in a while, so let’s see what’s new there, especially in the Google integration part….

Reborn Google Resources

Just before the KDE PIM sprint in Berlin this month, I’ve sat down and written completely new API for LibKGAPI (the library that implements Google API and is used by the Akonadi resources for Google services). The new API is job-based, and therefore much more awesome than the old one (which is known to suck). Anyway - what does this mean? It means that the new resources are awesome as well!

Google Contacts resource now has a full support for contacts groups. All contacts are stored in the top-level collection and are linked to the respective groups, so it does not matter where you edit the contact, you are still modifying the same instance. Like in the web interface.

Google Calendar now supports limited sync, so you can choose to only sync events from last year, or last two years (the default is last 3 years) instead of the full history.

Both resources have improved status reporting, error handling, are more stable (no more mystery crashes due to unhandled exceptions thrown from LibKGAPI) and subjectively synchronization is faster too.

Murdered Google Resources

As most of you probably noticed by now, Google is planning to shut down Google Reader by July 1. It’s pitty, because I already had a fully working Akonadi resource for Google Reader ready in the akregator_port branch. Cost me lot of time and nerves. Well, the resource is not there anymore and the only memory of it is greader branch with API implementation in LibKGAPI (which will die as well sooner or later). The good news however is that I can now help Alessandro and Frank with ownCloud News and the ownCloud Akonadi resource, so that we rock when Akregator2 is out :-) I can’t wait to see what has changed in ownCloud since I installed 3.0.0 some time ago…

Upcoming Google Resources<

I have two feature requests in bugzilla: one is to support Google Bookmarks, which is fairly complicated because of missing official API and absolutely no write API. So this is not going to happen soon. The second feature request is for Google Drive KIO slave. This is much more interesting task. I already tried writing Google Docs KIO slave about three years ago and I failed epically. Retribution! There’s almost complete API implementation by Andrius in LibKGAPI git, so I plan to port it to LibkGAPI2 and see whether we can together fight the Dark side and create a nice and shiny KIO slave.

Finally, deep in the dark corners of my mind, my so far the most evil plan is slowly shaping. The plan includes modifying the current IMAP resource, reusing most of it’s code and subclassing some specific parts to build a native GMail Akonadi resource that would support some GMail-specific IMAP extensions. The main idea is to support one-mail-in-multiple-folders-at-once case. Right now the IMAP resource handles that by creating a new instance of the same email in multiple folders. My bold plan is to store all emails in Inbox and link them to respective folders. This means that marking an email as read in one folder, will automatically mark it as read in all other folders (because it’s a single instance). The IMAP resource looks scary though, so I don’t know yet when I’ll get the courage (and time) to sit down and actually start coding…I guess probably after Akademy, after I talk to some people.

Batch Operations in Akonadi

I have talked to Volker Krause during the KDE PIM sprint about how to effectively handle “Mark feed as read” in the Google Reader resource. Currently, Akonadi creates a new notification for every change, therefore marking 300 items as read generates 300 notifications, which are delivered to the Akonadi resource, which should then create 300 HTTP request to store 300 changes. You probably agree that this slightly suboptimal. (I temporarily solved the problem by caching the notifications in the resource itself and then sending a big request to Google Reader at once). The solution that Volker suggested sounds fairly simple (it’s not) - batch notifications - i.e. a single notification about single change involving multiple items. The supported changes can be flags change, deleting or linking of items. By being able to deliver single notification about mass-change to Akonadi clients and to Akonadi resources represents new possibilities for optimizations. For instance the IMAP resource could simply send a single command to add a flag to multiple emails at once, instead of doing it one by one. The same goes for other operations and other resources that are dealing regularly with operations on larger sets of items. The obvious result: performance boost! After two weeks the work is in semi-working state - it works, but it goes nuts if more than 5 items are involved. The cause is known, but solution not (but I’ll get there eventually :-) )

Akregator 2

I’m occasionally helping with Akregator2 (Akonadi port of Akregator). Recently (ok, it was two months ago… ) I’ve written Akonadi Nepomuk Feeder plugin that is feeding RSS Articles into Nepomuk and a Search window (slightly inspired by the one in KMail) in Akregator2 where you can do full-text search (+ search via other criteria, including author’s name and date of publishing) based on data indexed in Nepomuk. Obviously, when I wanted to demo that on the KDE PIM sprint I found out that it’s not working as good as I thought, so there’s still some work to be done. But in general I’m happy to say, that from time to time it finds something :-).

Ok, so that’s about what I was, am and will be working on in KDE PIM. Here I’d like to say big thank you to all KDE PIM devs, because they are doing an incredible job. Thank you!

Hi! Nearly four weeks after the [0.3](https://dvratil.cz/2012/04/akonadi-google-0.3-arrives/ release of Akonadi resources for Google there’s a new version with just a few, but important bug fixes and improvements.

UPDATE: In order to use the Google resources, you either need KDE 4.9 (which the resources are part of), or you need to install LibKGAPI 0.4.1 (or newer) and Akonadi resources from git://anongit.kde.org/scratch/dvratil/akonadi-google-resources. See this discussion for details.

Fixed bugs and crashes:

• Bug #296541 - Uncaught exception in signal handler in Contacts resource
• Bug #297824 - Uncaught exception in signal handler in Calendar resource
• Bug #297548 - Crash at akonadi start after having added a new contact - resource
• Bug #298054 - Can’t build libkgoogle with KCal
• Bug #298518 - Unable to edit newly created events
• Bug #298519 - Deleting events incorrectly reports an error

The first two bugs were especially tricky as I couldn’t reproduce them, but many users were affected by ugly and repeating crashes. But now the “Google experience” is much much better :).

Big thanks go to Alex Fiestas who has contributed various improvements to libkgoogle to better work with 3rd party apps (so now we can be looking forward to his web-accounts wizard :) ).

Sources: akonadi-google-0.3.1.tar.gz (MD5: *fed8d9082547835ab916edd219831cf6)

Bye!

PS: I found this on Akademy wiki, so:

After many months of “I will release it next week” I finally released libkgoogle 0.3 and new version of Akonadi resources for Google this week.

So, what’s new? I managed to implement everything I described in [this post]( {% post_url 2011-11-25-akonadi-google-resource-whats-comming %} “Akonadi Google Resource: what’s coming?”) back in November. That’s support for multiple Google accounts, and merging the tasks resource into the calendar resource (so now it’s called “Calendar and Tasks resource”). The calendar now properly supports events recurrence and partially exceptions in recurrent events (there’s still some work to be done). The contacts resource now splits your contacts to “My Contacts” and “Others” groups. I hoped to fully support contact groups, the code was even in place, but I’ve run to some problems how to store it in Akonadi and unfortunately KAddressBook is not “compatible” with the Google’s concept of contact groups, so I decided to stick with the two elementary groups and hopefully I’ll get to this later (maybe some PIM dev could help me on Akademy? ;) )

If you run to any problems or bugs, please report them to the libkgoogle product in bugzilla.

Finally, I’d like to thank to Jan Grulich and Vojtěch Zeisek for putting their contacts and events at risk to test the pre-release versions and provided valuable feedback.

Sources

(Updated tarball!) akonadi-google-0.3.tar.gz (md5: 8c5c1e015068bea90bf25dd7858dc913)

If you want to follow the most recent development, you can use sources from the master branch.

Have a nice day!

It’s been a while since my last blog about the Akonadi Google resources and since my last contribution to the project (except for a few minor bug fixes lately). Today I decided to change it (at least the former) and show you that the project is still active (I just don’t have much time to work on it) and reveal to you what big changes I plan to do in the near future.

There are two questions people are often asking me: if it is possible to support multiple Google accounts and why are calendars and tasks split to two resources. If you are one of those people, I have good news for you: next release will support multiple Google accounts and the Tasks resource will be merged to the Calendars resource.

Because words are plain, I will rather show you some pictures of how I imagine the resources could look like.

The first image is the new preferences dialog for Calendar resource. In the first list, there will be list of all accounts. The list will be common for all resources, so you will see the same accounts in the Contacts resource as well. Below is the list of calendars for the selected account (the list will auto-update itself when you choose another account + there will be something like “Reload” button to update the list by hand) with possibility to create, modify or remove existing calendars not just from Akonadi, but from Google server as well. The last list view is for task lists.

The second image shows how the calendar editor could look like. The are two things I want to change yet: replacing the wide color combo box by a single button which will popup another dialog with list of colors to pick from (according to docs Google supports only limited set of colors for calendars) and I’d like to have the timezone list displayed as a combobox instead of a tree view. The editor for task lists will look similar.

Regarding editor of accounts, clicking on “Create” will simply display the Google login page, same as it does now when you click on “Authenticate” in the resource preferences dialog. Given second thought, the “Edit” button does not make much sense since you can’t change anything (maybe just the name of the account), “Remove” will just revoke the authentication.

And finally, on the last image you can see how the list of calendars in KOrganizer could look like. Each account would have sublist of calendars and tasks lists. The image is very inaccurate (copy&paste in Gimp :)), only calendars you have selected in the “Calendars” list in the preferences dialog will be displayed in the list and there will be task lists displayed as well (again, only the task lists you choose in the preferences dialog).

So, do you like it? :) I don’t have much time now and collage exams are closing in, but I hope to find some time during Christmas and as I know myself I will be hacking this new features all January instead of learning for the exams :)

And just to list some improvements since last time: we now support categories for calendar events, information about contact group membership is synchronized (you can’t yet move contacts between Google’s contact lists, but at least the resource does not automatically remove your contacts from the lists) and some issues with timezones and daylight saving were fixed.

And finally (I almost forgot about this) some more good news, especially for Fedora users: I’ve created a project in OpenSUSE Build Service, so you can add a repository to yum to have access to fresh Akonadi Google resources :) I’m will rebuild the packages after every important git commit, so you will be still up-to-date.

Fedora 16: http://download.opensuse.org/repositories/home:/progdan/Fedora_16/

Fedora 15: http://download.opensuse.org/repositories/home:/progdan/Fedora_15/

The project itself is available here: https://build.opensuse.org/package/show?package=akonadi-google&project=home:progdan

Fell free to modify the .spec file to support other distributions as well, I don’t have much skills in packaging .specs for anything else but Fedora :)

And if you just want to build the project from sources, you can get them from akonadi-google repo in KDE git: git clone git://anongit.kde.org/akonadi-google

Cheers,

Dan

UPDATE: In OpenSUSE, you can find snapshots in the KDE:Unstable:Playground repository: https://build.opensuse.org/package/show?package=akonadi-google&project=KDE:Unstable:Playground

Hi!

As the title says, I just added support for Google Tasks by creating the Akonadi Google Tasks Resource. The Tasks API provided by Google is really simple and does not support many properties, only name, summary, due to date, completed date and status. You can’t set progress percentage, start date, attendees nor reminders (this sucks!). Despite the fact, that the API provides means for tree-like structure of tasks (tasks and subtasks), it does not seem to work. So you can only have a linear list of tasks. A positive thing is, that due to this limited functionality of Google Tasks the resource has a full support of this API.

The reason for independent resource is that you can have multiple task lists in Google Calendar, thus merging this functionality into Google Calendar Resource is not an option. Unfortunately, you will now have the tasks resources displayed in the list of calendar resources in KOrganizer.

The second and very important update is, that we now have a product in KDE Bugzilla, so you don’t have to report bugs in comments here. The product is “libkgoogle” so you can report bugs or whishes here: https://bugs.kde.org/enter_bug.cgi?product=libkgoogle&format=guided.

If you think you have posted a request/report under an earlier blogpost and I still haven’t responded/fixed/implemented your request, please report it again in the bugzilla.

Sorry, no screenshots today :)

Bye!

Hi there,

today I finally pushed a completely new version of the Akonadi Google Resource. Since last update cca 2 months ago I’ve completely rewritten the resources, moving most of the functionality into libKGoogle. The library now provides access to Google services and can be easily used by any other project.

So what’s new? A lot of things changed for programmers. I’ve begun to fulfill my threats about providing a C++/Qt library for accessing Google Services. It’s not pure Qt, I’m using some KDE stuff in there, but that will change :)

LibKGoogle now provides API similar to QNetworkAccessManager (KGoogleAccessManager) through which you can send KGoogleRequest and if it works, you’ll receive a KGoogleReply. Each Google Service is represented by a KGoogleService subclass, implementing XML and/or JSON parser and serializer and providing informations about fetch/create/update URLs. Each object (contact, event, calendar) is represented by KGoogleObject subclass. For now, Event class is implements KCalCore::Event as well, the Calendar and Contact classes provide their own implementation of everything. Porting it to KABC/KCalCore is on TODO.

libKGoogle provides unified authentication framework and stores the tokens in KWallet. For now, you can only use one account per service (means that you can’t have two calendars from two different Google accounts), but this is on my TODO as well. The framework is also able to automatically refresh the tokens when they expire, so that should fix the problem with “resources forgeting the authentication”.

The API needs some more polishing and I must somehow force myself to write documentation, but the I think the libKGoogle can provide a solid base for any Google-related services.

The library is still in the same git repository with the Resources. I will try to move it to it’s own repo when it gets support for more services and the API will really stabilize.

From users’ point of view, there is just a very slight change in Contact Resource settings dialog. There are no new features in any resource. Though there are probably many new bugs :F So please try it out and don’t be afraid to spam me with any issue, idea, problem, comment, invitations to a beer etc…:)

With libKGoogle I hope more developers will begin to create Google-related applications for Qt and KDE. This library could one day be a good start-point for them.

Plans for future versions:

• support more contacts and event properties
• support multiple Google accounts per service
• changing contact photos
• tie KGoogleObjects to KCalCore and KABC
• API cleanup
• DOCUMENTATION :/
• make the libkgoogle optionally KDE-independent (long-term plan)

It’s been more then two weeks since my last update about progress on Akonadi Google Resource, so here we go. Many bugs in calendar were fixed, resource now supports creating and editing events and Contacts resource can fetch contact photos.

So let’s start with the contacts. I’ve implemented fetching photos from Google Contacts, so now you can see all your contacts smiling at you from your address book. Updating/removing photos is on to-do, but there are more important things to be implemented/fixed first.

Now to calendar resource: you can now finally create and edit events from KOrganizer. I’ve also fixed some issues with timezones, recurrence and reminders, so it really seems to work pretty good now. What has to be done are timezones in recurrence. I’ve also implemented removing of events, but there seems to be some hidden problem somewhere, because Akonadi refuses to invoke itemRemoved() signal, thus informing the resource, that event was removed and the resource can’t send deletion request to Google Calendar….I hope I’ll get to look into this this week. Of course, if you feel you can do something about it, just send patches ;)

Known bugs: except for the broken deleting of events there is a problem with authentication. Google only remembers the last authentication token it has issued, so when you have two Calendar resources for example, Google will accept requests only from the one you added as second. When you re-authenticate the first calendar, Google will forget access token of the second calendar and vice versa. Using KWallet to store the access token would be probably the best way how to share a single token among multiple resources and of course it would increase security a lot, but I need to learn how to do it first :).

And finally, about future of this project…I was in contact with developer of the original Akonadi GData resource (now called Akonadi GCalendar & Contacts resources) and we agreed that it would be nice to have a C++/Qt library for full support of GData API (including other services, not just calendars and contacts). I will be slowly moving all the GData-related code from resources to a separate library. My idea is to provide a pure-Qt library with optional KDE extensions (like conversion to KABC or KCalCore objects), which could be enabled during build-time, so that the library could be KDE-(in)dependent, depending on developers’ needs. But this will be content of an another blog post, maybe later when things will begin to move. Now is my priority to improve the resource, because the Akonadi-related code is minimum compared to GData-related code which can be later dragged into the library.

So, that would be all for now, please test the resource and give me some feedback :)

Cheers,

Dan

Hi,

so it’s been almost ten days since first version of Akonadi Google Resource and now here comes 0.2. This version introduces Google Calendar Resource, so that you can finally access Google Calendar from comfortable interface of KOrganizer.

UPDATE: Akonadi Google Resource is now in KDE git repository!

As of now the resource supports read-only access to calendars, so you can just watch your busy schedule, but you still have to update it in GMail. Some basic write support is the main goal for version 0.3.

A good news is, that the resource supports multiple calendars, so you can finally see all your calendars in KOrganizer, not just the default one. You have to add a new resource for each calendar you want to access. In configuration dialog you can then choose, which calendar you want to sync with.

The resource supports, except for the basic properties, multiple attendees, including their roles and types, recurrent events (but without exceptions) and reminders (popup-dialog and email notification).

As I already said, in 0.3 I’d like to introduce read-write access to Google Calendar and add photo-fetching to the Contacts resource.

If you want to give it a try, you can download 0.2 tarball from here, or you get the most recent sources from git:

git clone git://gitorious.org/akonadi-google/akonadi-google.git

Bye!

UPDATE: Akonadi Google Resource is now in KDE git repository. To fetch it, use: git clone git://anongit.kde.org/akonadi-google.

My job as a Red Hat internet is development of Evolution, the groupware software. Although I’m using Gnome 3 and all the Gtk stuff in work, I’m still loyal to KDE on my personal laptop. But one thing I really like on Evolution and I really miss in KDE PIM is proper support for Google services. As result of this and after some talking with my flatmate, I started to work on real, fullfeatured Akonadi Resource for Google Services.

At this moment, after about week of work I have working Contacts Resource. It can fetch, create, update and remove contacts from Google server using their GData API. The implementation of GData protocol is far from complete, as of now only few basic values are supported (name, emails, phones, addresses, notes).

Since this implementation is actually “working”, I’ve decided to release it as 0.1.

Version 0.2 should bring basic support for Google Calendar and most probably some major refactoring of libkgoogle. Following releases will focus on improving support of GData protocol, including fetching of contacts photos. I have to mention here, that the code is inspired by code of the great Akonadi Facebook Resource, because this is my first Akonadi-related code and I really need something to learn from :)

There are no screenshots worth posting at this moment, you rather try checking out the sources, maybe sending some patches back, what do you say? ;) Sources are available in Gitorious: http://www.gitorious.org/akonadi-google, clone URL is git://gitorious.org/akonadi-google/akonadi-google.git.

UPDATE: the project is now hosted in KDE Git repo on *https://projects.kde.org/projects/playground/pim/akonadi-google.