Lately there were some issues with the Google integration in Kontact which
caused that it is no longer possible to add new Google Calendar or Gmail account
in Kontact because the log in process will fail. This is due to an oversight on
our side which lead to Google blocking Kontact as it did not comply with Google’s
policies. We are working on resolving the situation, but it will take a little
bit.
Existing users should not be affected by this - if you already had Google
Calendar or Gmail set up in Kontact, the sync should continue to work. It is
only new accounts that cannot be created.
In case of Gmail the problem can mostly be worked around when setting up the
IMAP account in KMail by selecting PLAIN authentication1 method in the
Advanced tab and using your email and password. You may need to enable Less
Secure Applications in your Google account settings in order to be able to log
in with regular email address and password.
If you are interested in the technical background of this issue, the problem
comes from Google’s OAuth App Verification process. When a developer wants to
connect their app to a Google service they have to select which particular
services their app needs access to, and sometimes even which data within each
service they want to access. Google will then verify that the app is not trying
to access any other data or that it is not misusing the data it has access to -
this serves to protect Google users as they might sometimes approve apps that
will access their calendars or emails with malicious intent without them
realizing that.
When I registered Kontact I forgot to list some of the data points that Kontact
needs access to. Google has noticed this after a while and asked us to clarify
the missing bits. Unfortunately I wasn’t able to react within the given time
limit and so Google has preemptively blocked login for all new users.
I’m working on clarifying the missing bits and having Google review the new
information, so hopefuly the Google login should start working again soon.
Despite its name, the PLAIN authentication method does not weaken the
security. Your email and password are still sent safely encrypted over the
internet. ↩︎
I’m sorry to bring bad news, but after trying to fight some last minute bugs in the new Gmail resource today, I realized that pushing the resource into KDE Applications 4.14 was too hurried, and so I decided not to ship it in KDE Applications 4.14. I know many of you are really excited about the Gmail integration, but there are far too many issues that cannot be solved this late in 4.14 cycle. And since this will probably be the last 4.x release, shipping something that does not perform as expected and cannot be fixed properly would only be disappointing and discouraging to users. In my original post I explained that I was working on the Gmail integration to provide user experience as close as possible to native Gmail web interface so that people are not tempted to switch away from KMail to Gmail. But with the current state of the resource, the effect would be exactly the opposite. And if the resource cannot fulfil its purpose, then there’s no point in offering it to users.
Instead I will focus on implementing the new native Gmail API and merging together the existing Google resources to create a single groupware solution that will provide integration with all Google’s PIM services - contacts, calendars, tasks and emails.
I already teased publicly about the new Gmail resource on Google+ yesterday, now it’s time for some more explanations and…screenshots!
What is this about?
A native Gmail Resource for Akonadi that will bring much better integration of Gmail features into Kmail.
I’ve been talking about it and promising it for quite some time. But now thanks to some changes to the regular IMAP Resource that Christian Mollekopf has done recently, I could finally start working on it!
But…why?
Those who use Gmail know that Gmail does some things differently than most normal mail services. The biggest difference is that there are not really any folders with emails. Instead there’s one folder with all your emails and then there are labels, that you can assign to emails and then you can just filter your emails by the labels. And you can assign multiple labels to one email.
Yeah, but why bother? It already works quite well with the normal IMAP resource, right?
Yes, Gmail is able to hide this specialities from regular email clients so that they can still work with Gmail like with any other generic mail server, but at the cost of losing some features.
More and more often you can hear today that desktop email clients are dead and the future is in webmail (and cloud). And for many users who only have one email account this is true - why having KMail and KOrganizer etc. running, when they can have all this and more opened constantly in a single tab in their web browser? And the truth is, that Gmail is simply the largest mail provider in the world today. So if we want to persuade all these users to keep using KMail, we need to provide a user experience that is as close as possible to the native web interface. And for that we need a native Akonadi Resource ;-)
(Note: I’d like to avoid flamewars about “desktop clients are not dead vs. are dead” - I believe they are not dead for people who use more than one email account. They will cling to desktop clients until the dawn of the Gods, and even longer, but for normal users with just one mail account, it might be just matter of years to leave desktop clients. But who knows. impossible to see the future is).
Ok, so what’s the difference between Gmail and IMAP Resources?
The Gmail Resource supports some Gmail-specific IMAP extensions. In other words, it can speak and understands Gmail’s IMAP dialect. This means that the Gmail resource can handle the Gmail specifics better than the generic IMAP Resource:
Flattened folder hierarchy. This is best shown on screenshots: on the left, there’s a folder hierarchy as shown in KMail when using the current IMAP resource, on the right there’s the same account but synced via the new Gmail resource.
One email to rule them all! As explained above, one email on Gmail can have multiple labels. But when you sync the mail via normal IMAP, you get a copy of that email in each folder. That means, that marking the mail as read will only mark as read that copy, but not the other copies in other folders, simply because KMail does not know they are effectively the same mail.
The Gmail resource is aware of this, an it syncs all your emails into one hidden folder and then just links them to the actual folders representing Gmail labels that you can see in KMail, so when you mark an email as read in any folder, it will mark it as read in all folders it’s linked into. Awesome, right?
OAuth authentication. The regular IMAP resource only supports the regular username-password authentication (and GSSAPI), which means that your password is stored in the computer somewhere, and if you use 2-step verification, you need to generate an app-specific password.
The Gmail resource has support for Gmail’s OAuth2, so you only enter your password once into Google’s web login, and the resource will then use a special tokens issued by Gmail with limited life-span to authenticate all your requests.
The authentication is actually powered by LibKGAPI, a Qt/KDE library that implements various Google APIs, so it has the same look and uses the same code as the Akonadi Resources for Google Calendars and Google Contacts.
(Funny story and a question: I actually had to write a custom plugin for Cyrus-SASL to support XOAUTH2 mechanism, as upstream does not support it. Does anyone know whether there’s an existing implementation somewhere on the interwebs that I could use instead my crappy plugin?)
Simpler configuration. This is not really that big, bacause you don’t open the dialog very often, but I really like the configuration dialog a lot: simple and without complex options like encryption, mechanisms… This is simply because Gmail supports only a specific set of IMAP features, so I could just remove lot’s of stuff from the configuration dialog making it thinner and much easier to understand (IMO).
Push-notifications for all folders. The IMAP Resource can only monitor one folder for changes (using IMAP IDLE), because of certain technical restrictions. The folder usually is the Inbox. So if you have server-side filtering, you will never get push-notifications about new emails arriving to your other folders.
The limitation for watching only one folder applies to the Gmail Resource too. However since we understand Gmail, we can watch the “All Mail” folder, instead of the Inbox, so this way we get push notifications about emails from absolutely all folders (except for Trash and Junk folders, but who cares about these). Thinking about it, I could even remove the “Check Interval” option from configuration now.
Uhm ok, so what’s the state of the resource?
Currently the resource is still in a branch, waiting for some more features to be finished and for Christian to approve some of my changes to the IMAP resource (I’ll bribe him with some beer during Akademy, if necessary ;-)), and some changes must be done in KMail to properly support copying and moving of the linked emails, but other than that, it already works quite nicely :-)
The KDE PIM Sprint is over (unfortunately…I could do this every day :-)), so now it’s time for some recap of what has been done. I’ll try to cover the Akonadi side, and leave the rest up to others to cover their projects ;-)
Akonadi Server optimizations
We finished and reviewed Volker’s old branch with a big optimization of the database schema. On my computer it reduces size of the file with the largest table by 30% and it speeds up all queries on that database, because the WHERE condition now has to perform only integer comparision, instead of string.
This however means, that we have to migrate user’s database on start. During the migration it is not be possible to use any Akonadi-based applications. We improved the code so that the migration takes about 10 minutes on my computer (used to take 20 and more). I personally think that it’s acceptable “downtime” for a one-time migration, so after I finish testing the migration code on other backends, I’ll merge the branch to master and we’ll ship it with KDE 4.13.
There's always time for a beer!
Server-side Search
When using online IMAP, only headers are in Akonadi, the body is downloaded on-demand when the message is opened in KMail. This means that Nepomuk can’t index these emails and thus can’t include them in search results. To fix this case, we want to make use of IMAP’s SEARCH functionality. We simply ask Nepomuk to search it’s database of indexed emails, at the same time we send IMAP server the same search query and then we just merge results and show them to users. Most of the infrastructure in Akonadi Server has been in place for a long time now, so I’ll just undust it, adopt it to our current needs and we should be good to go ;-)
Using Akonadi to store tags
Working on tags!
I already mentioned this in the previous report: we want to cache tags in the Akonadi database and write them to storage backends if they support it (for instance as additional flags to emails on IMAP server, as CATEGORIES into events in iCal, etc.). Thanks to it it will be possible to share tags between multiple computers, yay! We just need to modify the Nepomuk libraries, so that when you ask Nepomuk for all data tagged with “Holiday”, Nepomuk can search it’s own database and also query Akonadi. Another benefit will be that filtering emails in KMail by tags will be much much faster, because the relation will be stored locally in Akonadi and we won’t have to talk to Nepomuk, which is very slow (mostly because of Virtuoso).
[…] KMail has the second best threading in the world, I think, second only to mutt because that is faster. […]
Why can’t KMail just have the very best threading in the world? Because right now it has to fetch the entire folder from Akonadi in order to be able to perform Subject comparision when building threads. That’s both very slow and CPU-intensive operation. So we thought: let’s store information about relations between emails in Akonadi, and when KMail asks for content of a folder, we give back only first few conversations just to fill the screen, and then fetch remaining conversations on demand when user scrolls down. This should make opening even massive folders superfast and should save a lot of memory, too.
Akonadi and KDE Frameworks
Hopefully David's code is more stable than his towers :)
The most-awaited discussion of the entire sprint was about KDE PIM and KDE Frameworks. When should we start? What has to be done? What can we use this opportunity for? From Akonadi point of view I want to do several things: remove deprecated API, change some API so that we use consistent naming and separate UI and non-UI stuff. Volker Krause suggested that we could move the client libraries into Akonadi repository with Akonadi server, so thatwe could share some of the code (protocol parsers for example), which I like, so we’ll go for that, too.
A bit unrelated, but still: the Akonadi server already compiles with Qt 5 for a while, so possibly during this development cycle we might switch to supporting only Qt 5 (and making use of all the C++11 awesomeness). There’s a little library that kdepimlibs link against, so we just build both Qt 4 and Qt 5 versions of it. Akonadi depends only on QtCore and QtDBus, so we only need distros to ship qt5-qtbase, which we believe most of them do by now.
Gmail resource
Vishesh and Àlex (Photo by Lukáš Tinkl)
I’ve been promising this for ages, now I finally discussed this with others, got some input and can start hacking :-) Let’s see if I can do something before Christmas ;-) Gmail resource would store all your mails in one folder and would create virtual folders for each label and just link emails from the “All mails” folder into respective labels. This way the emails will share flags (read/unread), and you will even be able to manage the labels by linking or unlinking emails from label folders in KMail.
Here I’d like to thank everyone for coming to Brno - if was a lot of fun and great pleasure to see all of you again, and also thank Red Hat for letting us use the office. Looking forward to see you all again on FOSDEM, next sprint, Akademy or anywhere else :-)
I haven’t blogged about my involvement in KDE PIM in a while, so let’s see what’s new there, especially in the Google integration part….
Reborn Google Resources
Just before the KDE PIM sprint in Berlin this month, I’ve sat down and written completely new API for LibKGAPI (the library that implements Google API and is used by the Akonadi resources for Google services). The new API is job-based, and therefore much more awesome than the old one (which is known to suck). Anyway - what does this mean? It means that the new resources are awesome as well!
Google Contacts resource now has a full support for contacts groups. All contacts are stored in the top-level collection and are linked to the respective groups, so it does not matter where you edit the contact, you are still modifying the same instance. Like in the web interface.
Google Calendar now supports limited sync, so you can choose to only sync events from last year, or last two years (the default is last 3 years) instead of the full history.
Both resources have improved status reporting, error handling, are more stable (no more mystery crashes due to unhandled exceptions thrown from LibKGAPI) and subjectively synchronization is faster too.
Murdered Google Resources
As most of you probably noticed by now, Google is planning to shut down Google Reader by July 1. It’s pitty, because I already had a fully working Akonadi resource for Google Reader ready in the akregator_port branch. Cost me lot of time and nerves. Well, the resource is not there anymore and the only memory of it is greader branch with API implementation in LibKGAPI (which will die as well sooner or later). The good news however is that I can now help Alessandro and Frank with ownCloud News and the ownCloud Akonadi resource, so that we rock when Akregator2 is out :-) I can’t wait to see what has changed in ownCloud since I installed 3.0.0 some time ago…
Upcoming Google Resources<
I have two feature requests in bugzilla: one is to support Google Bookmarks, which is fairly complicated because of missing official API and absolutely no write API. So this is not going to happen soon. The second feature request is for Google Drive KIO slave. This is much more interesting task. I already tried writing Google Docs KIO slave about three years ago and I failed epically. Retribution! There’s almost complete API implementation by Andrius in LibKGAPI git, so I plan to port it to LibkGAPI2 and see whether we can together fight the Dark side and create a nice and shiny KIO slave.
Finally, deep in the dark corners of my mind, my so far the most evil plan is slowly shaping. The plan includes modifying the current IMAP resource, reusing most of it’s code and subclassing some specific parts to build a native GMail Akonadi resource that would support some GMail-specific IMAP extensions. The main idea is to support one-mail-in-multiple-folders-at-once case. Right now the IMAP resource handles that by creating a new instance of the same email in multiple folders. My bold plan is to store all emails in Inbox and link them to respective folders. This means that marking an email as read in one folder, will automatically mark it as read in all other folders (because it’s a single instance). The IMAP resource looks scary though, so I don’t know yet when I’ll get the courage (and time) to sit down and actually start coding…I guess probably after Akademy, after I talk to some people.
Batch Operations in Akonadi
I have talked to Volker Krause during the KDE PIM sprint about how to effectively handle “Mark feed as read” in the Google Reader resource. Currently, Akonadi creates a new notification for every change, therefore marking 300 items as read generates 300 notifications, which are delivered to the Akonadi resource, which should then create 300 HTTP request to store 300 changes. You probably agree that this slightly suboptimal. (I temporarily solved the problem by caching the notifications in the resource itself and then sending a big request to Google Reader at once). The solution that Volker suggested sounds fairly simple (it’s not) - batch notifications - i.e. a single notification about single change involving multiple items. The supported changes can be flags change, deleting or linking of items. By being able to deliver single notification about mass-change to Akonadi clients and to Akonadi resources represents new possibilities for optimizations. For instance the IMAP resource could simply send a single command to add a flag to multiple emails at once, instead of doing it one by one. The same goes for other operations and other resources that are dealing regularly with operations on larger sets of items. The obvious result: performance boost! After two weeks the work is in semi-working state - it works, but it goes nuts if more than 5 items are involved. The cause is known, but solution not (but I’ll get there eventually :-) )
Akregator 2
I’m occasionally helping with Akregator2 (Akonadi port of Akregator). Recently (ok, it was two months ago… ) I’ve written Akonadi Nepomuk Feeder plugin that is feeding RSS Articles into Nepomuk and a Search window (slightly inspired by the one in KMail) in Akregator2 where you can do full-text search (+ search via other criteria, including author’s name and date of publishing) based on data indexed in Nepomuk. Obviously, when I wanted to demo that on the KDE PIM sprint I found out that it’s not working as good as I thought, so there’s still some work to be done. But in general I’m happy to say, that from time to time it finds something :-).
Ok, so that’s about what I was, am and will be working on in KDE PIM. Here I’d like to say big thank you to all KDE PIM devs, because they are doing an incredible job. Thank you!
This is just a short note mostly for packagers and other developers using LibKGoogle (if there are any). Due to possible violation of Google trademark, the LibKGoogle library has been renamed to LibKGAPI. Thanks to Clifford Hansen for inventing such a nice name :)
Also the license has been changed and the project is now under GPLv2+ instead of GPLv3+.
The entire API is now encapsulated in KGAPI namespace instead of KGoogle and headers were moved to %prefix%/include/libkgapi.
All these changes are now reflected in LibKGAPI 0.4.0 release, which does not contain any other improvements or fixes. I decided to go directly to 0.4 instead of continuing the 0.3 series, as I consider this a big and important change.
The Akonadi resources for Google services in kdepim-runtime will depend on LibKGAPI 0.4.0, the patch has been already submitted for review.
The git repository is still called libkgoogle, it will be renamed when it’s moved to KDE extragear (or somewhere else). Stable branch is LibKGAPI/0.4.
Hi! Nearly four weeks after the [0.3](https://dvratil.cz/2012/04/akonadi-google-0.3-arrives/ release of Akonadi resources for
Google there’s a new version with just a few, but important bug fixes and improvements.
UPDATE: In order to use the Google resources, you either need KDE 4.9 (which the resources are part of), or you need to install LibKGAPI 0.4.1 (or newer) and Akonadi resources fromgit://anongit.kde.org/scratch/dvratil/akonadi-google-resources. See this discussion for details.
Fixed bugs and crashes:
Bug #296541 - Uncaught exception in signal handler in Contacts resource
Bug #297824 - Uncaught exception in signal handler in Calendar resource
Bug #297548 - Crash at akonadi start after having added a new contact - resource
Bug #298519 - Deleting events incorrectly reports an error
The first two bugs were especially tricky as I couldn’t reproduce them, but many users were affected by ugly and
repeating crashes. But now the “Google experience” is much much better :).
Big thanks go to Alex Fiestas who has contributed various improvements to libkgoogle to better work with 3rd party apps
(so now we can be looking forward to his web-accounts wizard :) ).
After many months of “I will release it next week” I finally released libkgoogle 0.3 and new version of Akonadi resources for Google this week.
So, what’s new? I managed to implement everything I described in [this post]( {% post_url 2011-11-25-akonadi-google-resource-whats-comming %} “Akonadi Google Resource: what’s coming?”) back in November. That’s support for multiple Google accounts, and merging the tasks resource into the calendar resource (so now it’s called “Calendar and Tasks resource”). The calendar now properly supports events recurrence and partially exceptions in recurrent events (there’s still some work to be done). The contacts resource now splits your contacts to “My Contacts” and “Others” groups. I hoped to fully support contact groups, the code was even in place, but I’ve run to some problems how to store it in Akonadi and unfortunately KAddressBook is not “compatible” with the Google’s concept of contact groups, so I decided to stick with the two elementary groups and hopefully I’ll get to this later (maybe some PIM dev could help me on Akademy? ;) )
If you run to any problems or bugs, please report them to the libkgoogle product in bugzilla.
Finally, I’d like to thank to Jan Grulich and Vojtěch Zeisek for putting their contacts and events at risk to test the pre-release versions and provided valuable feedback.
It’s been a while since my last blog about the Akonadi Google resources and since my last contribution to the project (except for a few minor bug fixes lately). Today I decided to change it (at least the former) and show you that the project is still active (I just don’t have much time to work on it) and reveal to you what big changes I plan to do in the near future.
There are two questions people are often asking me: if it is possible to support multiple Google accounts and why are calendars and tasks split to two resources. If you are one of those people, I have good news for you: next release will support multiple Google accounts and the Tasks resource will be merged to the Calendars resource.
Because words are plain, I will rather show you some pictures of how I imagine the resources could look like.
The first image is the new preferences dialog for Calendar resource. In the first list, there will be list of all accounts. The list will be common for all resources, so you will see the same accounts in the Contacts resource as well. Below is the list of calendars for the selected account (the list will auto-update itself when you choose another account + there will be something like “Reload” button to update the list by hand) with possibility to create, modify or remove existing calendars not just from Akonadi, but from Google server as well. The last list view is for task lists.
The second image shows how the calendar editor could look like. The are two things I want to change yet: replacing the wide color combo box by a single button which will popup another dialog with list of colors to pick from (according to docs Google supports only limited set of colors for calendars) and I’d like to have the timezone list displayed as a combobox instead of a tree view. The editor for task lists will look similar.
Regarding editor of accounts, clicking on “Create” will simply display the Google login page, same as it does now when you click on “Authenticate” in the resource preferences dialog. Given second thought, the “Edit” button does not make much sense since you can’t change anything (maybe just the name of the account), “Remove” will just revoke the authentication.
And finally, on the last image you can see how the list of calendars in KOrganizer could look like. Each account would have sublist of calendars and tasks lists. The image is very inaccurate (copy&paste in Gimp :)), only calendars you have selected in the “Calendars” list in the preferences dialog will be displayed in the list and there will be task lists displayed as well (again, only the task lists you choose in the preferences dialog).
So, do you like it? :) I don’t have much time now and collage exams are closing in, but I hope to find some time during Christmas and as I know myself I will be hacking this new features all January instead of learning for the exams :)
And just to list some improvements since last time: we now support categories for calendar events, information about contact group membership is synchronized (you can’t yet move contacts between Google’s contact lists, but at least the resource does not automatically remove your contacts from the lists) and some issues with timezones and daylight saving were fixed.
And finally (I almost forgot about this) some more good news, especially for Fedora users: I’ve created a project in OpenSUSE Build Service, so you can add a repository to yum to have access to fresh Akonadi Google resources :) I’m will rebuild the packages after every important git commit, so you will be still up-to-date.
Fell free to modify the .spec file to support other distributions as well, I don’t have much skills in packaging .specs for anything else but Fedora :)
And if you just want to build the project from sources, you can get them from akonadi-google repo in KDE git:
git clone git://anongit.kde.org/akonadi-google
As the title says, I just added support for Google Tasks by creating the Akonadi Google Tasks Resource. The Tasks API
provided by Google is really simple and does not support many properties, only name, summary, due to date, completed
date and status. You can’t set progress percentage, start date, attendees nor reminders (this sucks!). Despite the fact,
that the API provides means for tree-like structure of tasks (tasks and subtasks), it does not seem to work. So you can
only have a linear list of tasks. A positive thing is, that due to this limited functionality of Google Tasks the
resource has a full support of this API.
The reason for independent resource is that you can have multiple task lists in Google Calendar, thus merging this
functionality into Google Calendar Resource is not an option. Unfortunately, you will now have the tasks resources
displayed in the list of calendar resources in KOrganizer.
The second and very important update is, that we now have a product in KDE Bugzilla, so you don’t have to report bugs in
comments here. The product is “libkgoogle” so you can report bugs or whishes here:
https://bugs.kde.org/enter_bug.cgi?product=libkgoogle&format=guided.
If you think you have posted a request/report under an earlier blogpost and I still haven’t responded/fixed/implemented
your request, please report it again in the bugzilla.
