I want to introduce a new method of delivering change notifications from Akonadi server to clients inspired by IMAP IDLE extension.

+

===== Rationale =====

+

Even with batch notifications and server-side notification filtering, Akonadi is still flooding DBus with large amount of change notifications. When a notifications is delivered to clients' Akonadi::Monitors, they need to query Akonadi for the actual item, which generates additional IO/CPU load and slows down the process.

+

+

===== Implementation =====

+

Instead of DBus notifications, each Akonadi::Monitor would open a dedicated session, called IDLE session and push it's notification filter configuration to the server. The Akonadi server would then send change notifications via this session instead of using DBus messages. Unlike with DBus notifications, when additional roundtrip from Monitor to Server is needed to request the actual changed Item, Akonadi IDLE notification will, in addition to change description, already contain the actual Item.

+

+

This will make a use of the fact, that Server knows and has the Item (or Collection) at the moment when notification is generated, so it can easily serialize it (once) and send it to all interested parties.

+

+

I expect notable performance improvements and decrease in database load. The protocol-based "notifications: are also easier to extend, unlike DBus messages, which would solve our problem with having to introduce a new NotificationMessageVX every time we need to add something.

+

+

=== Server-side Change Recording ===

+

* By: Dan Vrátil

+

* Planned for: 4.14 / Akonadi 1.13

+

I want to move change recording from Agents to the Akonadi server. This feature depends on Akonadi IDLE and paves the path for addition features, like on-demand start/stop of Agents.

+

===== Rationale =====

+

Currently each Agent records change notifications into it's own file. This duplicates some data (not much, but still) on the disk, but also generates a lot of IO, because we need to sync() it every time, even when the notification stays there just for a brief moment. I propose we store the change notifications on the Akonadi server, which already has a storage suitable for them, but also sees the "bigger picture" of all Agents available and can optimize accordingly.

+

+

Additionally, with server-side change recording, we can shut down an Agent when it's not used or usable (think for instance IMAP resource when network connection is not available) and still record all it's notifications on the server, then dispatch them once the Agent is usable again. This on-demand start/stop of Agents however, is a different task/topic, which will probably not be implemented in time for 4.14 (probably a KF5 topic)

+

+

===== Implementation =====

+

Implementation of server-side change recording builds on top of Akonadi IDLE. The IDLE command will be extended to provide means for Akonadi::ChangeRecorder to enable/disable/suspend/resume change recording as well as request pending notifications from the server.

+

+

How the notifications will be stored on the Akonadi server has yet to be determined.

Roadmap / Planned Features

This section is for features where you have concrete plans for implementation (doesn't have to be in the next release).
A short description, the approximate timeline if you know already (e.g. for version X), and the reason why the feature is needed would be interesting.

New Kolab-Resource that derives from the IMAP-Resource

By: Christian Mollekopf

Planned for: 4.14

I plan to replace the kolab proxy with a new resource that derives from the imap resource (so no additional agent anymore). Each imap resource would have two root collections, one for email and one for groupware data.

Rationale

The primary reason is to reduce the overall complexity of the design, I expect the new resource to be easier to debug (because it's mostly the same codebase as the imap resource) and to be less error prone (because it's one less level of indirection).
Other reasons are:

improved performance: we can convert the objects directly inline, which saves processing power.

we have one configuration per account (as opposed to the kolabproxy), because each resource is only responsible for one account, allowing me to get rid of the per account configuration hacks in the kolabproxy.

A kolab account no longer requires two configuration dialogs, and the dialog can probably be simplified for kolab. (at least by default we only require server + login + password).

The groupware folders no longer need to be hidden in the email client because they are not created anymore

Range Queries

By: Christian Mollekopf

Planned for: 4.14

It should be possible to request items by certain properties, such as a start and end date of an event, so not all items need to be loaded into memory.

Rationale

One of the largest performance limitations is that we currently have to load all items into memory in order to be able to select the ones we're interested in, since the akonadi server doesn't know the payload. We therefore require indexes of certain interesting properties that allow us to query by these properties.

Implementation

The problem is very similar to search, so it can probably be solved similarly. I'm not sure whether the indexing should happen in the baloo indexer, or in a preprocessor, since IMO fulltext indexing can be "eventually consistent" (5min delay wouldn't be a big problem), where these indexes are required to make the item available to the application (the event will not appear before the corresponding index and open query is updated).
Applications would "search" for the item-set, that would be exposed as virtual collection. So the calendar would i.e. search for "events of week 42", for which it would receive an id of a virtual collection that it can monitor. That will give the calendar only the event's it requires for display, including notifications for the range it's interested in only. As the user scrolls in the view, the range query needs to be modified and updated accordingly.

Random notes:

I assume incremental updates will be crucial for performing well with large datasets (for the scrolling).

The calendar should probably load the whole month if it's displaying only a week to have a buffer for the scrolling.

Relations

By: Christian Mollekopf

Planned for: 4.14

We want to be able to create relations between items in akonadi, so we can for instance relate a note to an email.

Rationale

A generic mechanism is needed to associate items, other than the existing tags (which are visible and have a name, etc.), and collections (similar to tags). The relations mechanism should provide 1:1 relations with a type.
Usecases: note-email, todo-email, ... (full mesh).

Implementation

Relation is triple (id, type, id)

relation types are stored in RelationTypeTable

RelationCreateJob( id, type, id)

ReleationDeleteJob (id, type, id)

ReleationFetchJob(id, type) => list of items

ItemFetchJob - include list of items IDs that are in releation of specified type(s) with the fetched item

specify type(s) in ItemFetchScope?

Resources should be notified as well, so relations can be stored on the server. However, a relation should only be added to the resource if both ends of the relation are part of the resource, and it should be removed again if one end of the relation is not in the resource.

Frameworks

Akonadi IDLE

By: Dan Vrátil

Planned for: 4.14 / Akonadi 1.13

I want to introduce a new method of delivering change notifications from Akonadi server to clients inspired by IMAP IDLE extension.

Rationale

Even with batch notifications and server-side notification filtering, Akonadi is still flooding DBus with large amount of change notifications. When a notifications is delivered to clients' Akonadi::Monitors, they need to query Akonadi for the actual item, which generates additional IO/CPU load and slows down the process.

Implementation

Instead of DBus notifications, each Akonadi::Monitor would open a dedicated session, called IDLE session and push it's notification filter configuration to the server. The Akonadi server would then send change notifications via this session instead of using DBus messages. Unlike with DBus notifications, when additional roundtrip from Monitor to Server is needed to request the actual changed Item, Akonadi IDLE notification will, in addition to change description, already contain the actual Item.

This will make a use of the fact, that Server knows and has the Item (or Collection) at the moment when notification is generated, so it can easily serialize it (once) and send it to all interested parties.

I expect notable performance improvements and decrease in database load. The protocol-based "notifications: are also easier to extend, unlike DBus messages, which would solve our problem with having to introduce a new NotificationMessageVX every time we need to add something.

Server-side Change Recording

By: Dan Vrátil

Planned for: 4.14 / Akonadi 1.13

I want to move change recording from Agents to the Akonadi server. This feature depends on Akonadi IDLE and paves the path for addition features, like on-demand start/stop of Agents.

Rationale

Currently each Agent records change notifications into it's own file. This duplicates some data (not much, but still) on the disk, but also generates a lot of IO, because we need to sync() it every time, even when the notification stays there just for a brief moment. I propose we store the change notifications on the Akonadi server, which already has a storage suitable for them, but also sees the "bigger picture" of all Agents available and can optimize accordingly.

Additionally, with server-side change recording, we can shut down an Agent when it's not used or usable (think for instance IMAP resource when network connection is not available) and still record all it's notifications on the server, then dispatch them once the Agent is usable again. This on-demand start/stop of Agents however, is a different task/topic, which will probably not be implemented in time for 4.14 (probably a KF5 topic)

Implementation

Implementation of server-side change recording builds on top of Akonadi IDLE. The IDLE command will be extended to provide means for Akonadi::ChangeRecorder to enable/disable/suspend/resume change recording as well as request pending notifications from the server.

How the notifications will be stored on the Akonadi server has yet to be determined.

Brainstorming

List in this section ideas that you think should be implemented, but you currently have no plans of working on in the near future.
It will hopefully allow us to share some of the ideas that are currently only existing in one or the other head.

Job-based Resource interface

By: Dan Vratil, Christian Mollekopf

The idea is to replace the current ResourceBase interface with a job-based one. Each task (retrieveItems, itemChanged, collectionAdded, ....), is a job, that can be implemented by each resource implementation.

Rationale

This will result in a simpler interface (since the resources are doing an async operation anyways), make it easier to support parallellism of tasks, etc.

Other benefits are:

no more AgentBase::Observer - we can add new new "tasks" withouth having to create another ObserverV42

possible paralellization - multiple jobs can be running at the same time, speeding up sync for instance

much nicer and easier-to-use API

no more asserts because resource implementation called taskDone() when no task was running, etc....

Planned implementation

we introduce this as ResourceBaseV2 and deprecate ResourceBase

resources are slowly ported to ResourceBaseV2

initially the current ResourceScheduler can be used without almost any modifications

Or the resource injects a ResourceTaskFactory that creates the jobs, or would that again result in the ObserverV42 problem?

ResourceBase starts the Job like if it was dispatching a task

Testability

While creating ResourceBaseV2 we should also make it testable. See the IMAP-Resource how this could work.
Ideally all akonadi related jobs are part of the respective job/task interface, which makes it possible for the tests to test each individual job without requiring an akonadi server.

Calendar events envelopes

By: Christian Mollekopf

Extract and store certain data from events into PLD:ENVELOPE part. Allows MemoryCalendar to retrieval all ENVELOPEs and then fetch PLD:RFC822 only for events that are relevant to required data/time.

It first needs to be verified how large the impact of this would be, since calendar data is relatively small (except for attachments).

Extracted values

Envelope should contain

DTSTART (timestamp)

DTEND (timestamp)

RECURRENCENED( timestamp) - calculated in serializer

HASALARM (bool) => for korgac

SUBJECT (string)

=> Everything that is required to display the event in the calendar view.