This blog post is long overdue, but now that I'm back home from the KDE PIM Sprint in Toulouse, which took place last weekend, there's some more news to report.

Akonadi Improvements

On the sprint, I finally finished and merged a new improvement in Akonadi called Notification Payloads. I will not go into the technical details here, the most important thing is that this new improvement will notably reduce the CPU and disk load in Akonadi, especially during intensive operations like email sync. It should also help with the long-standing issue regarding errors and email duplication when using POP3 and local mail filters. Finally, this new feature opens doors to further improvements and optimizations like server-side change recording (technicalities here) and ultimately being able to
shut down Akonadi Resources when they are not needed and start them on-demand, thus saving some more resources.

As I was touching the internal notification system in Akonadi I also improved the relevant debugging tools in Akonadi Console, our developer and debugging tool for Akonadi. Based on input from Sandro I also added Logs view. Thanks to that it's now possible to see debug output from all running Akonadi applications straight in the Akonadi Console without the need to restart Akonadi or the application from the terminal to see the debug output. This will make it easier for users to provide us with relevant information to help us debug and solve their Akonadi issues.

Kontact Improvements

This was just a minor change, but it finally solved my long-standing issue with Kontact and Breeze: the side-pane icons to choose between different Kontact modules were colorful - the only non-monochromatic part of Kontact which was so obviously not fitting into the rest of the UI. With a tiny change, the icons are now also monochromatic, making the Kontact window look more uniform.

Native Gmail authentication for IMAP and SMTP

For a while now the IMAP resource supports logging into Gmail accounts using the so-called OAuth method, where you provide your credentials into the Gmail login window which also supports two-factor authentication. The IMAP Resources was forcing the OAuth method with Gmail for everyone, but this requirement has now been relaxed. Although the IMAP resource will choose this method by default it's possible now to also choose the traditional authentication methods like with any other email provider.

Secondly, the OAuth support has finally landed also into our SMTP module which is used for sending emails, so if you select this method in your Outgoing account configuration with Gmail, you no longer need to use "App-specific passwords" from Gmail.

Syndication Cleanup

The Syndication library is used to retrieve and parse RSS and ATOM feeds and is used among others by Akregator. We have now cleaned up the library and removed some redundant dependencies so that we will eventually be able to move it into KDE Frameworks so that even more applications can benefit from it.

Going to Windows

Thanks to a huge effort from Hannah we are now able to build Akonadi and other parts of the KDE PIM stack on Windows. While we are still a long way away from having Kontact properly running on Windows, we managed to get Akonadi to work on Windows with some other programs. Windows is a huge platform and Kontact with all its features and functionality could be a good competition to established PIM solutions there and a huge potential to grow our user and developer base. While we still focus primarily on Linux, we are slowly looking forward to extending our reach to Windows.

Bugfixes

A lot of them. Big thanks to David Faure who spent a big part of the weekend debugging his IMAP resource to figure out why it keeps getting stuck on occasions. He fixed several issues in the IMAP resource so that it properly reconnects after server connection is lost or times out (due to poor internet connectivity usually) and also found and fixed some issues in Akonadi syncing code.

Future

What's next then? We will continue to work towards a stable release for Windows,
and hopefully soon finish the rewrite of the indexing and search infrastructure
in KDE PIM to make it faster, reliable and more useful again. There's also a lot
of smaller tasks and improvements to look into during the year.

Back in August, we ran a survey to get input from our users and get a better understanding of how they use KMail. First, let me start by thanking everyone who took their time to fill in the survey. We collected over 3000 responses which is much more than we expected. Thank you very much! We got some interesting numbers and data from the survey, which I'll analyze later, but to my big surprise, the most interesting part was the comments that many of you left at the end of the survey. We got over 1000 comments which provided us with a consistent feedback from the userbase. In this and the next blog posts, I want to address the common themes, complaints, and remarks that appeared in the comments, address the concerns raised and present some action plans that we are going to take to address those.

What's going to happen to KMail when Kube is released

Nothing. Kube is not going to replace KMail. Both projects will co-exist and be continuously developed and improved. Each project targets a slightly different audience - KMail focuses on power users with several accounts and high volume email traffic, while Kube focuses on less advanced users. Kube will not, and is not attempting to, provide many of the advanced features that KMail has, instead, it will help KDE to provide a friendly PIM solution for all users.

"Akonadi should die"

We hear this a lot: "Akonadi sucks", "Akonadi should be replaced", etc. While we understand the frustration of many users who are having huge troubles with Akonadi, we are committed to it and we are convinced it is the right way to go. Akonadi got a lot of bad reputation in the early days, but we worked very hard to fix the bugs and improve the performance over the years and we will continue to do so in the future. I took some more detailed notes, so I have an idea what people dislike about it the most and will try to focus towards solving that. There also seems to be a lot of misconception as to what Akonadi really does and it is often blamed for things it's not directly responsible for. I might write a detailed blog post about that sometime in the future.

Some of you also mentioned Sink: Sink (formerly known as Akonadi-next, but no longer has anything to do with the current Akonadi) is the backend used by Kube. At this moment Sink is not mature enough and lacks the capabilities and features we need in order to be viable as a replacement for Akonadi. I personally like the concept and some of the ideas behind  Sink and I will try to get some of them into Akonadi and I hope that in the future we will be able to co-operate with Sink more closely.

Search is broken/useless

I was honestly surprised with the amount of complaints about the search feature of KMail - generally it boils down to searching not being reliable and returning wrong or no results. I took a good look into our indexing and searchimg code and indeed found numerous issues with the way we index data and query them. I already created a plan how to solve this and make search fast, reliable and giving more precise results. The work on this is already under way, the fixes to actual indexing and searching code will be available in the December release, with more high-level fixes (indexing speed, better search UI, result presentation etc.) coming next year.

Account management

This issue has been on our radar for some time now and it's good to know that it's something you want us to target and fix as well. Adding a new email account to KMail is super hard, you have to basically do three things - create the incoming email account, create an outgoing email account, create the identity, and then you need to re-visit each of the of the accounts to configure them. We have the Account Wizard (Settings -> Add account..., or Tools -> Account Wizard in older KMail versions) which simplifies adding new accounts, but it's not fully there yet and adjusting a configuration later on is still a painful experience spread across three different dialogs and many mouse clicks.

I took a look at what Thunderbird and Evolution do, and I also looked at K9, an opensource email client for Android, which is very close to KMail in the terms of account configurability (different identities per account, different outgoing accounts per identity etc.) and how their UI works. All in all, we have a lot of inspiration and examples to follow and build on, and we will work with the KDE VDG and our UX experts to overhaul the account management in KMail, making it much smoother and nicer experience.

Feature Discoverability

KMail has myriads of features, but many of them are well hidden from our users, almost as if we did not want anyone to use them. I've often seen comments like "It would be nice if KMail could do X" or "Until this survey, I did not even know KMail could do Y". So, we need to fix that too. There are several ways how to do that, I think the best one is a combination of all: we need to improve our UI and make features easily discoverable and ready at hand so that they are easy to use. Secondly, we need to bring our documentation up to date. Couple years ago Scarlett did a great job at updating our documentation, but there's still so much that is not covered. Finally, we need a good and easily accessible feature overview, ideally on a web page or wiki. We would welcome any help with writing and updating the documentation - this does not require any programming skills, just some free time and will to help :-) Get in touch with us on the kde-pim@kde.org mailing list!

More coming soon in part 2...

The main goal for me and Volker for this year Randa Meeting was to port KCalCore, our calendaring library, away from KDateTime. KDateTime/KTimeZone are classes for handling date, time and time zones which were used back in the KDE4 (and earlier) days when Qt's QDateTime wasn't good enough for us, mainly due to missing time zone support.

In Qt5 QDateTime finally gained all the necessary features we need which made it possible for us to leave KDateTime behind. But we couldn't just go and replace "K" with "Q" everywhere - there are many subtle (and not-so-subtle) differences in API, behavior and some missing features of QDateTime that required us to carefully consider each step. The fact that we started working on this over 2 years ago shows how challenging task this was.

We did not expect to finish the port here, but once we dived into it, things went fairly well and I'm glad to say that after four days of intensive hacking, surrounded by Swiss Alps and mountains of chocolate, we succeeded. KCalCore is now free of KDateTime and KTimeZone and that in turn made (after some minor adjustments) the rest of KDE PIM free of KDELibs4Support. That's a huge milestone for us.

Many thanks to John Layt for laying the initial ground for the porting and to Mario and Christian for the steady stream of chocolate and sweets to soothe our nerves :-)

If you want to help us to continue improving Kontact and other parts of KDE, please donate to the Randa fundraiser so that we can continue organizing similar productive sprints in the future. Thank you!

Getting started with contributing to KDE PIM can be hard - we have nearly 60 repositories with complicated dependencies - just getting that right can discourage many people from even trying. And then there's, of course, the risk factor of running development build alongside your production Kontact, endangering your precious emails.

To address all these issues I have created a Docker image. It's based on the KDE Neon Developer edition and it has all the dependencies pre-installed and pre-configured and comes with a set of handy shell scripts to make your life easier. It also has the environment set up properly so that you can run the development build of Kontact inside of the container - completely isolated from your production installation.

Interested now? Follow the instructions how to build the Docker image and how to run the container on our KDE PIM Docker wiki page.

The section regarding GPU drivers is still incomplete, if you have any knowledge regarding running OpenGL-enabled applications inside a Docker container with Nouveau or AMD/ATI drivers, let us know, please!

You can probably use our Docker image to develop other KDE applications in there as well, or you can take a look at Jos' blog about Developing KDE with Docker and create your own Docker image to work on your favorite KDE application.

Happy hacking!

Randa Meetings is an annual meeting of KDE developers in a small village in Swiss Alps. The Randa Meetings is the most productive event I ever attended (since there's nothing much else to do but hack from morning until night and eat Mario's chocolate :-)) and it's very focused - this year main topic is making KDE more accessible.

Several KDE PIM developers will be present as well - and while we will certainly want to hear other's input regarding accessibility of Kontact, our main goal in Randa will be to port away from KDateTime (the KDE4 way of handling date and time in software) to QDateTime (the Qt way of handling date and time). This does not sound very interesting, but it's a very important step for us, as afterward, we will finally be free of all legacy KDE4 code. It is no simple task, but we are confident we can finish the port during the hackfest. If everything goes smoothly, we might even have time for some more cool improvements and fixes in Kontact ;-)

I will also close the KMail User Survey right before the Randa meetings so that we can go over the results and analyze them. So, if you haven't answered the KMail User Survey yet, please do so now and help spread the word! There are still 3 more weeks left to collect as many answers as possible. After Randa, I will be posting a series of blog posts regarding results of the survey.

And finally, please support the Randa Meetings by contributing to our fundraiser - the hackfest can only happen thanks to your support!

You can read reports from my previous adventures in Randa Meetings in 2014 and 2015 here:

• Hacking my Way through Randa (2014-08-15)
• KDE PIM in Randa(2015-08-13)

Akademy is over and so here's a short summary of what the PIMsters have worked on during the past week.

Wiki Cleanup

Me and Volker sat down and went through all KDE PIM wikipages on community.kde.org, userbase.kde.org and techbase.kde.org. Most of our wiki pages are horribly outdated, so we tried to clean them up, remove pages that are no longer relevant or useful. With fewer pages to take care of and better overview of what all content we have, we should be able to keep them more up-to-date than we did in the past years.

Developer Story

Contributing to KDE PIM is hard and we know that. Getting all the dependencies and environment set up correctly is not trivial, and you can't run stable and development Kontact alongside each other easily.

We decided to address all those issues and make contributing to KDE PIM substantially easier. We are working on a Docker image that has all the dependencies and environment set up, so developers just need to run a single command to build entire KDE PIM. And thanks to the containerization, it's also possible to use the development version of Kontact in parallel with the stable version.

We hope that having ready-made development environment it will be easier for new contributors to get involved with KDE PIM. We will post a more details once the Docker image is ready.

Kontact Homepage

Right now kontact.kde.org and kontact.org just redirect to our page on the Userbase wiki. We decided that we want a simple, but professionally-looking web site to market Kontact as an actual product so that it appears more attractive to new users, especially those who will be coming from Windows in the future and contains comprehensive information for both users and developers.

KMail User Survey

During QtCon 2016 we started working on KMail user survey to get a better idea of what our user base is like, how they use KMail and what their impressions of it are. And now the survey is finally live, so please go and fill it if you haven't done so yet.

Wayland Support

Volker has finished porting Kontact to Wayland, so if you have Qt 5.9, you can now run Kontact natively on Wayland. Our main limitation was Wayland support in QWebEngine, which we use to render emails, but that has been resolved in Qt 5.9.

Merging Exchange Support

Krzysztof Nowicki has been working on Microsoft Exchange support for Kontact for a while now. We now have plans to merge his code into kdepim-runtime repository, so if everything goes right the Exchange support will be available out-of-the-box to all our users starting with the December release of KDE Applications.

Next Sprint

We will be meeting soon again in Randa. Our main plan for the sprint is to continue with removal of KDateTime from our code, and thus making KDE PIM free of kdelibs4support.

There's some more that I did not mention here, you can check the full notes for details.

Sounds like déjà vu? You are right! We used to have Facebook Event sync in KOrganizer back in KDE 4 days thanks to Martin Klapetek. The Facebook Akonadi resource, unfortunately, did not survive through Facebook API changes and our switch to KF5/Qt5.

I'm using a Facebook event sync app on my Android phone, which is very convenient as I get to see all events I am attending, interested in or just invited to directly in my phone's calendar and I can schedule my other events with those in mind. Now I finally grew tired of having to check my phone or Facebook whenever I wanted to schedule event through KOrganizer and I spent a few evenings writing a brand new Facebook Event resource.

Inspired by the Android app the new resource creates several calendars - for events you are attending, events you are interested in, events you have declined and invitations you have not responded to yet. You can configure if you want to receive reminders for each of those.

Additionally, the resource fetches a list of all your friend's birthdays (at least of those who have their birthday visible to their friends) and puts them into a Birthday calendar. You can also configure reminders for those separately.

The Facebook Sync resource will be available in the next KDE Applications feature release in August.

If you can't synchronize a folder in KMail and you are seeing "Multiple Merge Candidates" error after the synchronization fails, here's a how to fix the folder to make it synchronize again - basically you force KMail to forget and re-sync the folder again.

1. Open Akonadi Console.
2. Go to the Browser tab.
3. Right-click the broken folder and select "Clear Akonadi Cache" - this will remove all emails from the folder in Akonadi. This will NOT delete your emails on the server.
4. Akonadi Console will freeze for a while, wait until it unfreezes (sorry, it's just a developer tool, we don't have a very good UX there :-)).
5. Logout and login to make sure all PIM components are restarted.

After login start KMail (or Kontact) and hit "Check mail". KMail will now re-download all emails from the previously broken folder. This may take a while depending on how large the folder is and how fast your internet connection is. After that the synchronization should work as expected.

In the upcoming KDE Applications 16.12.1 release Akonadi will have a fix that fixes the reason why the "Multiple Merge Candidates" error occurs, so hopefully in the future you should not see this error anymore.

It's actually been over for two days, but I'm still sitting in Berlin and only now got to write something.

As every year, it was great to see all my friends and fellow hackers again. Thanks everyone for being so awesome, I enjoyed every day of QtCon and Akademy with you. Can't wait to meet everyone again next year :-)

In the terms of KDE PIM, this year's Akademy was very productive. We had our KDE PIM BoF session on Monday afternoon, where we spent most of the time discussing KDE PIM User Survey - a plan of mine to get more information about our users and their use cases. The results will help us, the KDE PIM devs, to better understand how our users use our software and thus prioritize our focus. We ended up with an initial set of questions we intend to ask our users and next week I'll meet with some more KDE PIM hackers that could not attend Akademy and we will finalize the set of questions so that we can publish the survey later this month.

We also talked about some other topics on the meeting, like releasing of some of our libraries that Kube wants to use and so on.

You can read the mostly complete meeting notes on the KDE PIM wiki.

Outside of the BoF session we touched the topic of KDE PIM sprints and meetings. We want them to be more focused in the future, i.e. having a specific topic for each meeting that we will all work on together. We hope to do one meeting in Autumn this year to finish porting KCalCore away from KDateTime and KDELibs4Support, then a Spring meeting in Toulouse (which has become our new regular place for Spring sprints), then Randaaaaaaaaaaa (which gives us full 6 days of uninterrupted hacking with only small breaks to eat Mario's chocolate :-)) and then it's Akademy time again!

Oh and I can't forget to mention that the KDE PIM team was awarded the Akademy Award for our work on, well, KDE PIM :-). It was a great feeling to stand on the stage knowing that people appreciate our work.

---

Regarding my PIM work during Akademy, I think this year was pretty good. I did my share of partying during QtCon, so I could spent most of Akademy days hacking from morning until they kicked us out from the venue, and then continuing with some more hacking in the KDAB office until late night. Already before the event I merged a big change that improves the Akonadi change notification system. I managed to polish it during Akademy and fix several crashes and bugs.

Another big change was to our test-suite. It contains among other things integration tests, where we run an actual Akonadi server in an isolated environment (so that it does not touch any real data) and test whether clients interact with it as they are supposed to do. For these integration tests we've been only using the SQLite database until now, but I have now enabled MySQL and PostgreSQL too, so we run each test three times - once for each of the backends. This has revealed several corner-case issues that we weren't aware of until now. The test still run into some issues on the CI on build.kde.org but locally they pass for me (with only one exception). Addressing those issues is on the top of my todo list now.

I also started working on an experimental XML->C++ generator, which would allow me to get rid of some 12,000 lines of hand-written C++ code that implements the communication protocol between Akonadi server and the clients. Instead I will generate the code from a simple XML. So far I managed to get it to generate a code that compiles, but there's still a lot of work ahead to make it generate an optimal and correct code.

I'll spend the next week meeting all my colleagues from KDAB, which I'm really looking forward to. Although I know many of them from KDE, there are lots of people I haven't met yet, so it will be great to attach faces to the names. After that, it's back to Prague and to regular work (and some more Akonadi hacking ;-)).

Oh and if you haven't heard yet, KDE is celebrating 20th birthday. Go check out the timeline of KDE and get the amazing "20 Years of KDE" book!

If you want to know what we did in KDE PIM in the last year and what we are planning to do and achieve in the next one come to my KDE PIM Status Report talk on Sunday at 1 PM. If you want to get into more technical details and discussions about KDE PIM there will also be a KDE PIM BoF session on Monday afternoon.

See you in Berlin!