Search form

SyncEvolution 1.3.99.3 released

So far the development cycle for SyncEvolution 1.4 mostly focused on implementing the "PIM Manager" D-Bus API for IVI use cases [1]. The 1.3.99.3 pre-release starts to include more features and bug fixes again for syncing. For example, several ActiveSync improvements from Graham Cobb were included.

The remaining goal for 1.4, besides more testing of course, is to work out how to support Google CalDAV and CardDAV. I am in discussion with Google to get SyncEvolution whitelisted for use with these APIs - fingers crossed...

Upgrading from release 1.2.x

The sync format of existing configurations for Mobical (aka Everdroid)
must be updated manually, because the server has encoding problems when
using vCard 3.0 (now the default for Evolution contacts):
syncevolution --configure \
syncFormat=text/x-vcard \
mobical addressbook

Upgrading from releases before 1.2

Old configurations can still be read. But writing, as it happens
during a sync, must migrate the configuration first. Releases >= 1.2
automatically migrates configurations. The old configurations
will still be available (see "syncevolution --print-configs") but must
be renamed manually to use them again under their original names with
older SyncEvolution releases.

Changes 1.3.2 -> 1.3.99.3

PIM Manager: add ReplaceSearch, always allow it

The new ReplaceSearch is more flexible than RefineSearch. It can
handle both tightening the search and relaxing it. The downside of it
is the more expensive implementation (must check all contacts again,
then find minimal set of change signals to update view).

Previously, a search which had no filter set at all at the begining
could not be refined. This limitation of the implementation gets
removed by always using a FilteredView, even if the initial filter is
empty.

PIM Manager: introduce CreateConfig()

That SetPeer() allows modifying and creating a config leads to race
conditions when multiple clients want to create a config. The new
CreateConfig() avoids that by atomically checking that a config does
not exist yet and creating it.

SetPeer() is still available for backwards compatibility. It continues
to be used for modifying an existing config in TestContacts.testSync
to check the effect of the logging settings.

PIM Manager: fix double entries in filtered search with limit

Stressing the FilteredView by using it in tests originally written
for the FullView showed that the filling up a view may have
used data while it was inconsistent internally, leading to
contacts being present multiple times.

If phone number search is enabled in EDS, then the direct search in
EDS now uses the more accurate
E_BOOK_QUERY_EQUALS_NATIONAL_PHONE_NUMBER comparison, with the E164
formatted caller ID as value to compare against. This gives
semantically correct results. The previous solution (now the
fallback) had to use substring searches, which did not match if the
contact's phone number was not formatted according to E164 and
which may have matched the wrong contacts if the trailing numbers
are the same.

When available, the pre-computed E164 number from EDS will be used
instead of doing one libphonebook parser run for each telephone
number while reading. Benchmarking showed that this parsing was the
number one hotspot, so this is a considerable improvement.

PIM Manager: fix error messages

Ensure and check that no unnecessary ERROR messages are printed.
libfolks was used slightly incorrectly, leading to several
harmless error messages (glib asserts). libphonenumber printed
its error messages to stdout.

PIM Manager: fix memory leaks during writing of contacts

Constructing the GValues created additional references instead
of taking over ownership as intended.

D-Bus server: fix read-after-free bug when using syslog

openlog() expects the string to remain valid. Must ensure that in
LoggerSyslog by making a copy. Found with valgrind.

PIM Manager: make implementation of some of the D-Bus methods thread-safe

The goal is to make it easier to extend syncevo-dbus-server
with other IPC mechanisms, which then can call the native C++
code directly.

That code was not prepared to handle calls in threads other than the
main one. Now this is checked when entering the methods and work is
shifted to the main thread if necessary. In the meantime the calling
thread waits for completion.

Changed the code to compile with gee 0.8, as used by folks 0.9.x.
Older versions of folks are no longer supported.

PBAP: support Bluez 5

The new Bluez 5 API is the third supported API for doing PBAP
transfers. It gets checked first, then the PBAB backend falls back to
new-style obexd (file based, similar to Bluez 5, but not quite the
same) and finally old-style obexd (data transfer via D-Bus).

In contrast to previous APIs, Bluez 5 does not report the reason for a
failed PBAP transfer. SyncEvolution then throws a generic "transfer
failed" error with "reason unknown" as message.

command line: recover from slow sync with new sync modes

The error message for an unexpected slow sync still mentioned
the old and obsolete "refresh-from-client/server" sync modes.
Better mention "refresh-from-local/remote".

CalDAV: more workarounds for Google CalDAV + unique IDs

Google became even more strict about checking REV. Tests which
reused a UID after deleting the original item started to fail sometime
since middle of December 2012.

Google CalDAV for a while (December 2012 till January 2013) sent
invalid XML back when asked to include CardDAV properties in a
PROPFIND. This got rejected in the XML parser, which prevents
syncing calendar data:

Without the explicit TYPE=INTERNET, email addresses sent to a Nokia
e51 were not shown by the phone and even got lost eventually (when
syncing back).

This commit ensures that the type is set for all emails sent to any
Nokia phone, because there may be other phones which need it and
phones which don't, shouldn't mind. This was spot-checked with a N97
mini, which works fine with and without the INTERNET type.

This behavior can be disabled again for specific Nokia phones by
adding a remote rule which sets the addInternetEmail session variable
to FALSE again.

Non-Nokia phones can enable the feature in a similar way, by setting
the variable to TRUE.

SyncML: config option for broken peers

Some peers have problems with meta data (CtCap, old Nokia phones)
and the sync mode extensions required for advertising the restart
capability (Oracle Beehive). The default in SyncEvolution is to
advertise the capability, so manual configuration is necessary when
working with a peer that fails in that mode.

Because the problem occurs when SyncEvolution contacts the peers
before it gets the device information from the peer, dynamic rules
based on the peer identifiers cannot be used. Instead the local config
must already disable these extra features in advance.

"noctcap" disables sending CtCap. "norestart" disables the sync mode
extensions and thus doing multiple sync cycles in the same session
(used between SyncEvolution instances in some cases to get client and
server into sync in one session).

Both keywords are case-insensitive. There's no error checking for
typos, so beware!

The "SyncMLVersion" property was chosen because it was already in use
for configuring SyncML compatibility aspects and adding a new property
would have been harder.

ActiveSync: added support for specifying folder names

Previously, the database field was interpreted as a Collection ID. This adds
logic to allow the database to be interpreted as a folder path. The logic is:

1) If the database is an empty string, pass it through (this is the most
common case as it is interpreted as "use the default folder for the
source type").
2) If the database matches a Collection ID, use the ID (this is the same as
the previous behaviour).
3) If the database matches a folder path name, with an optional leading "/",
use the Collection ID for the matching folder.
4) Otherwise, force a FolderSync to get the latest folder changes from the
server and repeat steps 2 and 3
5) If still no match, throw an error.

ActiveSync: support for listing databases

Now --print-databases scans folders on the ActiveSync server and
shows suitable folders for the ActiveSync backends instead of the
previous, hard-coded help text.

The command line swallowed errors thrown by the backend while listing
databases. Instead it just showed ": backend failed". The goal
was to not distract users who accidentally access a non-functional backend.
But the result is that operations like --configure or --print-databases could
fail without giving the user any hint about the root cause of the issue.

Now the error explanation in all its gory details is included.

For example, not having activesyncd running leads to:
INFO] eas_contact: backend failed: fetching folder list:
GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name
org.meego.activesyncd was not provided by any .service files

1.3.99.1 => 1.3.99.2, 13.12.2012

PIM Manager searches for a caller ID ('phone' search) in EDS
directly while folks still starts up. No unification is done of
these results. Intermediate results are replaced by the final ones
from folks once those are ready.

Useful for moving the session directories to a temporary file system.
They are essentially just useful for debugging when used as part of
PIM Manager.

"logdir" - a directory in which directories are created with
debug information about sync session

"maxsessions" - number of sessions that are allowed to exist
after a sync (>= 0): 0 is special and means unlimited,
1 for just the latest, etc.;
old sessions are pruned heuristically (for example,
keep sessions where something changed instead of
some where nothing changed), so there is no hard
guarantee that the last n sessions are present.

Avoid writing config file changes to disk by enabling a new
"ephemeral" mode for syncing via the PIM Manager. In this mode,
config file changes are not flushed resp. discarded directly.
This prevents writing to .ini files in ~/.config.

The "synthesis" binfile client files are still written, but they get
redirected into the session directory, which can (and should) be set
to a temp file system and get deleted again quickly.

Data dumps are turned off now in the configs created by the PIM
Manager.

syncevo-dbus-server: use syslog instead of standard output by default

syncevo-dbus-server: command line options for controlling
output and startup

Using the UID as part of file names gets more problematic when
allowing colons. Remove that character from the API and enforce
the format in the source code.

PIM Manager API: introduce contact ID and use it for reading

This makes it easier for a client to fully polulate its view with
contact data. Previously it could happen that due to concurrent
changes in the server, a client was returned data for the same
contact multiple times. A client had to detect that and re-issue
read requests.

The callback is guaranteed to be invoked once when a search has
finished sending its initial results, and not sooner. This makes it
possible to check whether the current data contains some contact or
not.

A 'limit' search term with a number as parameter (formatted as string)
can be added to a 'phone' or 'any-contains' search term to truncate the
search results after a certain number of contacts. Example:
Search([['any-contains', 'Joe'], ['limit', '10']])
=> return the first 10 Joes.

As with any other search, the resulting view will be updated if
contact data changes.

The limit must not be changed in a RefineSearch(). A 'limit' term may
(but doesn't have to) be given. If it is given, its value must match
the value set when creating the search. This limitation simplifies the
implementation and its testing. The limitation could be removed if
there is sufficient demand.

PIM Manager: fix refining a search

Due to not mapping the local index in the view to the parent's index,
refining only worked in views where parent and child had the same
index for the contacts in the search view.

PIM Manager: fix starting when done via search

When the unified address book (= FullView) was not running yet at the
time when a client wanted to search it, the unified address book was
not started and thus the search never returned results.

The problem was in the C++ D-Bus binding. If the method that gets bound
to D-Bus returns a value, that value was ignored in the signature:
int foo() => no out parameter

It works when the method was declared as having a retval:
void foo (int &result) => integer out parameter

This problem existed for both the libdbus and the GIO D-Bus
bindings. In SyncEvolution it affected methods like GetVersions().

PIM Manager performance: pre-compute normalized telephone numbers

Looking up by phone number spends most of its cycles in normalizing of
the phone numbers in the unified address book. Instead of doing that
work over and over again during the search, do it once while loading.

Looking up a phone number only once does not gain from this change, it
even gets slower (more memory intensive, less cache locality). Only
searching multiple times becomes faster.

Ultimately it would be best to store the normalized strings together
with the telephone number inside EDS when the contact gets
created. Work on that is in progress.

PIM Manager: improve performance of FullView sorting

This fixes the hotspot during populating the FullView content: moving
contacts around required copying IndividualData and thus copying
complex C++ structs and strings. Storing pointers and moving those
avoids that, with no lack of convenience thanks to boost::ptr_vector.

Reordering also becomes faster, because the intermediate copy only
needs to be of the pointers instead of the full content.

PIM Manager example: add benchmarking

The new "checkpoints" split up the whole script run into pieces which
are timed separately, with duration printed to stdout. In addition,
tools like "perf" can be started for the duration of one phase.

EDS: fix creating databases

--create-database was broken in combination with the final code in EDS
3.6 because it passed NULL for the UID to e_source_new_with_uid(),
which is considered an error by the implementation of that
method. Must use e_source_new() if we don't have a UID.

fixed some memory leaks, extended tests to cover new features and bugs

SyncEvolution 1.3.99.1, 25.10.2012

workarounds for warnings from g++ 4.5

engine: : local cache sync mode

This patch introduces support for true one-way syncing ("caching"):
the local datastore is meant to be an exact copy of the data on the
remote side. The assumption is that no modifications are ever made
locally outside of syncing. This is different from one-way sync modes,
which allows local changes and only temporarily disables sending them
to the remote side.

Another goal of the new mode is to avoid data writes as much as
possible.

This new mode only works on the server side of a sync, where the
engine has enough control over the data flow. Setting "sync" to:

"local-cache-incremental" will do an incremental sync (if possible)
or a slow sync (otherwise). This is usually the right mode to use,
and thus has "local-cache" as alias.

"local-cache-slow" will always do a slow sync. Useful for
debugging or after (accidentally) making changes on the local side.
An incremental sync will ignore such changes because they are not
meant to happen, aren't checked for to improve performance and
thus will leave client and server out-of-sync!

Both modes are recorded in the sync report of the local side. The
target side is the client and records the normal "two-way" or "slow"
sync modes.

With the current SyncEvolution contact field list, first, middle and
last name are used to find matches for contacts. For events, tasks
and memos, time, summary and description are used.

HTTP proxy: useProxy=0 overrides http_* env variables

Previously, if http_proxy was set, a proxy was used even if
explicitly disabled. This prevented disabling the use of a proxy
which only made sense in some cases, like accessing something
that runs locally. Explicitly telling SyncEvolution to ignore
http_proxy is necessary because it doesn't support no_proxy.

WebDAV: auto-discovery fix

With Google Contact + CardDAV the auto-discovery failed after
finding the default address book, without reporting that result.

command line: implement --create/remove-database

Creating a database is only possible with a chosen name. The UID is
chosen automatically by the storage. Only implemented in the EDS
backend.

file backend: sub-second mod time stamps

Change tracking in the file backend used to be based on the
modification time in seconds. When running many syncs quickly (as in
testing), that can lead to changes not being detected when they happen
within a second. Now the file backend also includes the sub-second part of the
modification time stamp, if available.

This change is relevant when upgrading SyncEvolution: most of the
items will be considered "updated" once during the first sync after
the upgrade (or a downgrade) because the revision strings get
calculated differently.

D-Bus server: avoid progress outside of 0-100% range

For example in the new TestLocalCache.testItemDelete100, the
percentage value in the ProgressChanged signal become larger
than 100 and then revert to 100 at the end of the sync.

Seems the underlying calculation is faulty or simply inaccurate.
This is not fixed. Instead the result is just clipped to the valid
range.

These binaries include the "sync-ui" GTK GUI and were compiled for
Ubuntu 10.04 LTS (Lucid), except for "syncevolution-activesync" which
depends on libraries in Debian Squeeze, for example EDS 3.4.

Older distributions like Debian 4.0 (Etch) can no longer be supported
with precompiled binaries because of missing libraries, but the source
still compiles when not enabling the GUI (the default).

The same binaries are also available as .tar.gz and .rpm archives in
the download directories. In contrast
to 0.8.x archives, the 1.x .tar.gz archives have to be unpacked and the
content must be moved to /usr, because several files would not be found
otherwise.