Archive for July, 2010

Over the last two months, we have been working with developers from the kernelhwsrv, mds and musicplayer packages to improve the performance of MTP synchronisation.

Media Transfer Protocol (MTP) is a standard originally developed by Microsoft that allows users to transfer music files (in our case from the PC to a Symbian device).

The music metadata such as the album name, artist, cover art etc. is stored in Symbian SQL so any performance improvements in persistentdata make a difference towards the end-user performance. I have summarised the changes relating to persistentdata below.

Other changes have been made in mds and musicplayer packages as well, e.g. tuning of database configuration parameters, but I haven’t described these.

Increased the soft heap limit to 8MB

Symbian SQL uses SQLite in shared-cache mode. This reduces the amount of memory and file I/O needed by the Page Cache because multiple connections to the same database share a single cache. In this mode, there is a single page cache for each open database.

The memory available for all page caches is limited by the soft heap limit: a limit enforced by the SQLite library on the total amount of heap memory allocated by SQLite. The limit is called a “soft” limit because SQLite does not guarantee the memory usage of the library will stay under this limit. SQLite tries to stay within the limit, but will still allocate memory if it is unable to find any memory to reuse. If SQLite has already reached the soft heap limit during a database transaction, it will remove unmodified pages from the page cache and write pages to disk to allow new pages to be read and updated.

The soft heap limit on Symbian^3 was originally set to 1024kB. During our investigation, we found that this was no longer an appropriate size considering the current usage of the database. For example, in MTP synchronisation, the thumbnail server writes batches of 60 thumbnails for the album art to its database. The total amount of data (metadata + BLOB) to be written in each batch is typically between 1.7MB and 2.5MB. This means that the whole transaction cannot be stored in memory before being flushed to disk.

When the soft heap limit is too low, the database commit will have a more random I/O pattern (it jumps back and forth in file position) because the database pages are written more than once to stay under the soft heap limit. This leads to bad performance especially on slower media like SD cards.

After some experimentation with batch sizes and soft heap limit configurations, we determined that the soft heap limit should be substantially increased. The default soft heap limit for Symbian SQL was increased from 1MB to 8MB for hardware targets in the week 21 delivery from Nokia (difference listing). This change reduced the time taken for MTP music download by 20%.

The soft heap limit for the WINSCW emulator configuration was not modified (left as 1MB). There is a limited virtual address space available for the emulator (as discussed on the forum) and increasing the soft heap limit means that our maximum EPOCHEAPSIZE would need to change too.

Increased the capacity of RFileBuf64 to hold 4 pages

All file access in Symbian SQL uses the RFileBuf64 class. The implementation has been updated to ensure that the buffer is always large enough to contain 4 pages from the database being accessed.

If 4 pages or more can fit into the default buffer size of 8kB, then the buffer size will not been modified. If this isn’t possible (e.g. because the database page size is 16kB) then the buffer is resized to contain 4 pages. The read-ahead is set as a multiple of the page size. This applies to the journal and main database file.

Reduced calls to RFile::SetSize made by RFileBuf64

The RFileBuf64 implementation was updated to defer calls to RFile::SetSize until the pages are going to be written to disk. This avoids redundant file metadata updates when pages are written out-of-order at the end of the database file.

Journal ‘resting size’ increased

Symbian SQL uses the SQLite PRAGMA journal_size_limit to limit the size of journal files after a database transaction has been committed. The ‘resting size’ for Symbian^3 journals before this change was 64kB.

One of my colleagues observed from a trace that committing a 60 thumbnail transaction took approximately 700ms, of which 200-300ms was FAT updates to allocate and deallocate clusters to the journal file. The thumbnail database journal typically grows to 150kB or more during each transaction, so a ‘resting size’ of 64kB results in cluster allocation/deallocation every time.

To avoid the repeated allocation and deallocation of FAT clusters on databases with larger page sizes, the resting size is set to 16 pages or 512kb (whichever is the smaller size). The minimum resting size remains 64kB.

Cluster allocation during file server cache write-back (kernelhwsrv)

Each bulk addition of thumbnails results in approx 2.5MB of new database pages. As the file server cache writes this to disk the FAT metadata is updated multiple times as clusters are allocated to the file. The file server cache implementation was modified to consolidate FAT cluster allocations during write-back.

I integrated a second contribution from NTT DOCOMO last week to fix a bug in one of the Symbian SQL test harnesses.

The bug (bug 3170) concerned the handling of end-of-line characters in an input file for the t_sqlperformance4 test. The input file contained a sequence of SQL statements executed during a Music Player application use case. These statements are parsed and executed by the test harness to measure performance.

The bug was only reproducible when the source code was obtained from Mercurial because the input file would have UNIX line-endings (line feed) rather than the traditional Windows line-endings (carriage-return followed by line feed). We didn’t spot the problem ourselves because the clients for our internal source code management systems automatically convert to Windows line-endings.

The problem is that the test assumed that the line endings for the text file would follow Windows conventions leading to the SQL statements being truncated when only line feed character was used. The fix allows both formats to be handled.

This type of problem was anticipated by William Roberts in May 2009. He wrote an excellent post about the handling of line-endings in Mercurial repositories called “A dilemma at the end of the line?” on the Symbian blog. It’s worth a read.

Thanks to NTT DOCOMO, INC for identifying this bug and contributing the solution to the community!