Archive for the ‘development’ Category

Over the last two months, we have been working with developers from the kernelhwsrv, mds and musicplayer packages to improve the performance of MTP synchronisation.

Media Transfer Protocol (MTP) is a standard originally developed by Microsoft that allows users to transfer music files (in our case from the PC to a Symbian device).

The music metadata such as the album name, artist, cover art etc. is stored in Symbian SQL so any performance improvements in persistentdata make a difference towards the end-user performance. I have summarised the changes relating to persistentdata below.

Other changes have been made in mds and musicplayer packages as well, e.g. tuning of database configuration parameters, but I haven’t described these.

Increased the soft heap limit to 8MB

Symbian SQL uses SQLite in shared-cache mode. This reduces the amount of memory and file I/O needed by the Page Cache because multiple connections to the same database share a single cache. In this mode, there is a single page cache for each open database.

The memory available for all page caches is limited by the soft heap limit: a limit enforced by the SQLite library on the total amount of heap memory allocated by SQLite. The limit is called a “soft” limit because SQLite does not guarantee the memory usage of the library will stay under this limit. SQLite tries to stay within the limit, but will still allocate memory if it is unable to find any memory to reuse. If SQLite has already reached the soft heap limit during a database transaction, it will remove unmodified pages from the page cache and write pages to disk to allow new pages to be read and updated.

The soft heap limit on Symbian^3 was originally set to 1024kB. During our investigation, we found that this was no longer an appropriate size considering the current usage of the database. For example, in MTP synchronisation, the thumbnail server writes batches of 60 thumbnails for the album art to its database. The total amount of data (metadata + BLOB) to be written in each batch is typically between 1.7MB and 2.5MB. This means that the whole transaction cannot be stored in memory before being flushed to disk.

When the soft heap limit is too low, the database commit will have a more random I/O pattern (it jumps back and forth in file position) because the database pages are written more than once to stay under the soft heap limit. This leads to bad performance especially on slower media like SD cards.

After some experimentation with batch sizes and soft heap limit configurations, we determined that the soft heap limit should be substantially increased. The default soft heap limit for Symbian SQL was increased from 1MB to 8MB for hardware targets in the week 21 delivery from Nokia (difference listing). This change reduced the time taken for MTP music download by 20%.

The soft heap limit for the WINSCW emulator configuration was not modified (left as 1MB). There is a limited virtual address space available for the emulator (as discussed on the forum) and increasing the soft heap limit means that our maximum EPOCHEAPSIZE would need to change too.

Increased the capacity of RFileBuf64 to hold 4 pages

All file access in Symbian SQL uses the RFileBuf64 class. The implementation has been updated to ensure that the buffer is always large enough to contain 4 pages from the database being accessed.

If 4 pages or more can fit into the default buffer size of 8kB, then the buffer size will not been modified. If this isn’t possible (e.g. because the database page size is 16kB) then the buffer is resized to contain 4 pages. The read-ahead is set as a multiple of the page size. This applies to the journal and main database file.

Reduced calls to RFile::SetSize made by RFileBuf64

The RFileBuf64 implementation was updated to defer calls to RFile::SetSize until the pages are going to be written to disk. This avoids redundant file metadata updates when pages are written out-of-order at the end of the database file.

Journal ‘resting size’ increased

Symbian SQL uses the SQLite PRAGMA journal_size_limit to limit the size of journal files after a database transaction has been committed. The ‘resting size’ for Symbian^3 journals before this change was 64kB.

One of my colleagues observed from a trace that committing a 60 thumbnail transaction took approximately 700ms, of which 200-300ms was FAT updates to allocate and deallocate clusters to the journal file. The thumbnail database journal typically grows to 150kB or more during each transaction, so a ‘resting size’ of 64kB results in cluster allocation/deallocation every time.

To avoid the repeated allocation and deallocation of FAT clusters on databases with larger page sizes, the resting size is set to 16 pages or 512kb (whichever is the smaller size). The minimum resting size remains 64kB.

Cluster allocation during file server cache write-back (kernelhwsrv)

Each bulk addition of thumbnails results in approx 2.5MB of new database pages. As the file server cache writes this to disk the FAT metadata is updated multiple times as clusters are allocated to the file. The file server cache implementation was modified to consolidate FAT cluster allocations during write-back.

The Symbian Bug Squad is looking for new members to find, locate and fix bugs in key areas of the Symbian Platform.

The next test day is on Monday (24th May) and it is not too late to get involved. The team will focus on the Organizer package (calendar, to-do, alarms, notes). More information is available from the forum announcement.

We intend to organize a persistentdata bug day in the near future. If you are interested to join us or have ideas what we should focus on, please let us know by adding a comment to this post or via the persistentdata-dev mailing list.

These steps explain what you need to do to set-up a development environment for the Symbian platform.

1. Install Perl
To build the code you need ActivePerl, a standard distribution of the Perl language. The Symbian Foundation recommend Perl 5.8.9. Further information in Kits Q&A.

2. Install Python
Symbian Build System v2 (SBSv2) needs ActiveState Python 2.5.1 or later installed. This is a standard distribution of the Python language. Symbian Foundation suggests Python 2.5.4, but I installed ActivePython 2.6.4.10 as the latest version of SBSv2 didn’t find python26.dll otherwise. Further information in Kits Q&A.

4. Install 7-zip
The Product Development Kit is distributed as a collection of zip files. You should install 7-zip from www.7-zip.org to unpack them because other zip tools have been known to have problems with the number of files in the archives.

You can download the zip files using your browser and unzip them using 7-Zip, or better still use the downloadkit.py utility as described in How to Download a PDK. Unzip everything to a new drive, created with the subst command.

Everything you need to know about building the code, working with Mercurial and making your first change is available from the Symbian Platform Contributors category on developer.symbian.org. If you get stuck, use the Developer Forums. We’ll be happy to help.

Summary
This post describes how you can install and configure a Symbian development environment for development. It might seem like a lot of steps to get set-up, but most of the tools are a one-off installation and once installed can be used for every new release of the platform.