There are issues with locking index files for KMail folders and mmap()/munmap() operations on Windows. Therefore, SQLite-based indices are in development. This page presents detailed development notes for this task.

There are issues with locking index files for KMail folders and mmap()/munmap() operations on Windows. Therefore, SQLite-based indices are in development. This page presents detailed development notes for this task.

−

Started: [[User:Jstaniek|jstaniek]] 11:35, 23 April 2008 (CEST)

+

Design and implementation started by [[User:Jstaniek|jstaniek]] 11:35, 23 April 2008 (CEST)

__TOC__

__TOC__

==Installation==

==Installation==

−

branches/work/kdepim-nommap branch has been created for SQLite mode, being synced with kdepim trunk. The only difference is kmail/ subdirectory. To build it on Windows:

+

Update (7 May 2008): commit 805075 merged changes related to the SQLite mode for KMail indices from /branches/work/kmail-nommap (r799390..804487) and /branches/work/kdepim-nommap/kmail (r804484..804960) back to trunk.

+

+

'''Thus, it is enough to use kdepim trunk now.''' Users of kdepim-nommap should execute ''emerge --unmerge kdepim-nommap'' and then ''emerge kdepim''.

+

+

Old instructions follow:

+

<strike>''branches/work/kdepim-nommap'' branch has been created for SQLite mode, being synced with kdepim trunk. The only difference is kmail/ subdirectory. To build it on Windows:

The last command will download the source code from the branch, with altered kmail/ source code, compile it and install. All the resulting files have the same names as in regular kdepim trunk. ''sqlite'' package will be also installed as a hard dependency.

+

The last command will download the source code from the branch, with altered kmail/ source code, compile it and install. All the resulting files have the same names as in regular kdepim trunk. ''sqlite'' package will be also installed as a hard dependency of the ''kdepim-nommap'' package.

+

</strike>

==Introduction==

==Introduction==

*we call the new implementation '''SQLite mode''' for short and the old implementation '''mmap mode'''

*we call the new implementation '''SQLite mode''' for short and the old implementation '''mmap mode'''

*SQLite 3.5.4 is used, as provided by emerge sqlite module; we should not allow using much older versions of sqlite, e.g. 3.1 because of file format differences

*SQLite 3.5.4 is used, as provided by emerge sqlite module; we should not allow using much older versions of sqlite, e.g. 3.1 because of file format differences

−

*we are using '''one''' .index.db file per account, not folder (NOTE: currently that's per folder...)

+

*we are using '''one''' .index.db "index file" per folder (on request, could be possible implement it per-account)

*kmailprivate links to sqlite library for SQLite mode, and KMAIL_SQLITE_INDEX is defined to enable #ifdef'd code

*kmailprivate links to sqlite library for SQLite mode, and KMAIL_SQLITE_INDEX is defined to enable #ifdef'd code

*kmfolderindex_sqlite.cpp is created and edited as a copy of kmfolderindex.cpp; kmfolderindex.cpp #includes kmfolderindex_sqlite.cpp for SQLite mode and then skips its own code completely

*kmfolderindex_sqlite.cpp is created and edited as a copy of kmfolderindex.cpp; kmfolderindex.cpp #includes kmfolderindex_sqlite.cpp for SQLite mode and then skips its own code completely

**'''See the page devoted to [[/merge|validation of the merege]]''' and the [[/merge#Results|results]].

==Open Questions==

==Open Questions==

Latest revision as of 09:35, 16 May 2008

There are issues with locking index files for KMail folders and mmap()/munmap() operations on Windows. Therefore, SQLite-based indices are in development. This page presents detailed development notes for this task.

Design and implementation started by jstaniek 11:35, 23 April 2008 (CEST)

Contents

Update (7 May 2008): commit 805075 merged changes related to the SQLite mode for KMail indices from /branches/work/kmail-nommap (r799390..804487) and /branches/work/kdepim-nommap/kmail (r804484..804960) back to trunk.

Thus, it is enough to use kdepim trunk now. Users of kdepim-nommap should execute emerge --unmerge kdepim-nommap and then emerge kdepim.

Old instructions follow:
branches/work/kdepim-nommap branch has been created for SQLite mode, being synced with kdepim trunk. The only difference is kmail/ subdirectory. To build it on Windows:

The last command will download the source code from the branch, with altered kmail/ source code, compile it and install. All the resulting files have the same names as in regular kdepim trunk. sqlite package will be also installed as a hard dependency of the kdepim-nommap package.

common code from {KMFolderMbox|KMFolderMaildir}::open( const char * ) moved to KMFolderIndex::openInternal()

common code from {KMFolderMbox|KMFolderMaildir}::create() moved to KMFolderIndex::createInternal()

2008-05-01..02

implemented updateIndex() (see the table) and added extended version of writeMessages() with WriteMessagesMode modes

writeMessages(): depending on the mode (WriteMessagesMode) the sql statement "INSERT INTO messages(data) VALUES(?)" or "INSERT OR REPLACE INTO messages(id, data) VALUES (?, ?)", thus unique IDs are created when new yet unsored messages are saved, for later use

as implementation of KMFolderIndex::indexLocation() is moved to FolderStorage (before FolderStorage only had it as abstract method); FolderStorage::idsLocation() and FolderStorage::sortedLocation() are added to avoid performing the math like mFolder->indexLocation() + ".sorted"

implemented using extended version of writeMessages() - with UpdateExistingMessages mode; still calls writeIndex() on writeMessages() failure and still (properly) does nothing if mDirty == false.

int KMFolderIndex::writeIndex( bool createEmptyIndex )

yes

creates db; creates tables messages table, insert messages to messages table, encoded as blobs using KMMsgBase::asIndexString(); header is not needed, but INDEX_VERSION is saved using PRAGMA user_version; byte order info is not saved: every integer is written using network order or as string

bool KMFolderIndex::readIndex()

yes

"SELECT id, data FROM messages" is called

int KMFolderIndex::count(bool cache)

yes

no changes

bool KMFolderIndex::readIndexHeader(int *gv)

bool KMFolderIndex::updateIndexStreamPtr(bool)

removed

removed

KMFolderIndex::IndexStatus KMFolderIndex::indexStatus()

yes

no changes so far; we have problems with trash folder being regenerated due to mtime of its .db file

void KMFolderIndex::truncateIndex()

yes

yes

recreate the db

void KMFolderIndex::fillMessageDict()

yes

no change as it just inserts messages from KMFolderIndex::mMsgList into KMMsgDict

KMMsgInfo* KMFolderIndex::setIndexEntry( int idx, KMMessage *msg )

yes

no change as it just sets creates a new KMMsgInfo object and inserts it into KMFolderIndex::mMsgList

bool KMFolderIndex::recreateIndex()

yes

no changes as it just calls createIndexFromContents() and readIndex()

off_t KMFolderIndex::mHeaderOffset

replace its public use (e.g. in KMFolderIndex::truncateIndex()) with additional bool indexOpened()