I have a fairly large library with 10k+ documents. Calibre takes a LONG time to load, easily more than 10 minutes after I open it to the time I can start using it. Is this normal, and is there any way to make it faster?

I have a fairly large library with 10k+ documents. Calibre takes a LONG time to load, easily more than 10 minutes after I open it to the time I can start using it. Is this normal, and is there any way to make it faster?

Hi SilverAce
Only have 2500 books in my library, but I did notice that adding custom columns with "instructions" to search each book entry and then do "something" definately could slow down Calibre startup.
Unfortunately that was more than a year ago, so I'm afraid I don't remember the specific details.
Have you tried starting Calibre from cmd using calibre-debug -g ?
As far as I remember this has been discussed before, with lots of tips to improve startup speed. Antivirus settings etc.
A bit more information could help us help you
Cheers, Per

Maybe the tips in this thread will help you speed calibre up.
You can use command line to start calibre easily in Windows, Linux and OS X. Note (from calibre's manual):

Quote:

On OS X you have to go to Preferences->Advanced->Miscellaneous and click install command line tools to make the command line tools available. On other platforms, just start a terminal and type the command.

I've always found it easiest just to go the Preferences option menu (select the drop down arrow besides Preferences) and select the Restart in Debug mode command or use the keyboard short cut for that (Ctrl+Shift+R).

You know what? I strongly suspect that the slowness is due to my external hard drive, a Western Digital My Passport, being formatted as NTFS and being accessed through a third-party driver. This benchmark shows that such drivers are FAR slower than HFS+ in dealing with folders with lots of files and subfolders.

You know what? I strongly suspect that the slowness is due to my external hard drive, a Western Digital My Passport, being formatted as NTFS and being accessed through a third-party driver. This benchmark shows that such drivers are FAR slower than HFS+ in dealing with folders with lots of files and subfolders.

So that may be the difference right there.

Another thing could be you might need to defrag your device. On Mac it is not needed but now that you say you are suing an ntfs portable device. I think you might need to defrag your harddrive

Another thing could be you might need to defrag your device. On Mac it is not needed but now that you say you are suing an ntfs portable device. I think you might need to defrag your harddrive

To do a defrag I think SilverAce will need to mount the drive on a Windows system. IMO defrag on NTFS has always been overrated as a solution to any problem, its a hang over from FAT and something with which to kick the can down the road - maybe Draghi should defrag the Euro

The 'drivers' from Paragon etc should be properly called called emulators. I use file system emulators on a Solaris system for NTFS and HFS+; performance of both is poor compared to native NFS over USB.

@SilverAce - if you've not done so already, I suggest you replicate the library to an HFS+ volume and point Calibre at it, if performance is OK then the problem is almost certainly the use of NTFS, although it could also be a bad drive, there may be OS/X utils available from WD to 'measure' drive health.

If you have access to a 'friendly' windows box, you could put Calibre Portable on it, and point it at the library on your WD USB drive - if its performance is acceptable then its 99.9 % sure its the NTFS on a Mac issue, if its slow then you may have a bad drive - WD have Windows utils for measuring drive health.

On reflection the data is being written to the NTFS volume via one of these so-called 'drivers', so a defrag may have a bigger effect than I would normally expect. Who knows how they manage the drive space.

Process Monitor for me shows lots of CreateFile, QueryNetwork, CloseFile entries, and when I look at the details it is requesting the Read Attributes of the files. It seems to do this for all books, so my startup is extremely slow.

I dont use any composite columns, so not sure why this is happening. I only have one custom Genre column. And although I have excluded the calibre directories, I've completely disabled my virus scanner whilst testing this to be sure it is not the cause.

If you have custom columns that depends on other columns, then I believe all values have to be initiated when you start Calibre. I have my metadata.db on a SSD drive and still noticed this very clearly. When I deleted those columns startup was fast again.

I have my metadata.db on SSD but the Calibre library on a NAS. I use a symlink in the Calibre library to the real file on the SSD, to speed up things. Calibre became a little bit more responsive that way.

Make sure you have enough memory in the computer.

You say you have no custom columns, so that may not be the problem, unless they are hidden?

I don't believe Calibre even access the actual books when starting. Only metadata.db.

If you have custom columns that depends on other columns, then I believe all values have to be initiated when you start Calibre. I have my metadata.db on a SSD drive and still noticed this very clearly. When I deleted those columns startup was fast again.

Not quite. The columns are evaluated only when displayed. Sorting on a column is equivalent to displaying all values for that column.

Quote:

I don't believe Calibre even access the actual books when starting. Only metadata.db.

Several versions ago calibre did look at attributes of books (modtime, etc) when starting. This was changed. I don't remember the version, but it was a couple of months ago at least. The current version does not look at book attributes unless you have a custom column that contains them (a "formats" column).

Several versions ago calibre did look at attributes of books (modtime, etc) when starting. This was changed. I don't remember the version, but it was a couple of months ago at least. The current version does not look at book attributes unless you have a custom column that contains them (a "formats" column).

I'm using the latest calibre 0.8.69 (portable to avoid any plugin issues) and it does seem to be reading attributes of all books in my library.

In order to test further I created a subset of my large > 10k book library. I need a subset to test because it's so long to open otherwise.

If I copy a subset of the library directory (eg only some of the books and the metadata.db), switch to the library, delete the one custom Genre column (just to be sure), and do a Database Maintenance->Restore Library, then when opening the resulting library, Calibre still read the attributes of all the books.

If however I create a new empty library, and then import the same subset of books into it, then when switching to that library, Calibre does not read the attributes of all the books, it only reads the first book attributes, and the next 6 book cover files.

So if I can figure out what the difference between the two test libraries is, I can work out where the issue lies. Does the first way of creating the subset test library still retain the preferences of the original library that are stored in the metadata.db? I'm guessing it does, and something in the preferences is causing Calibre to read all files.

The only code I can find in calibre that looks at the attributes of a format file is called when a template referring to a format is evaluated. Are you certain that you don't have a hidden composite column, and in particular a "formats" column? Or perhaps you have installed a plugin that walks through your books?

Copying a library folder will keep many of the preferences, and in particular all the custom columns, hidden or not. There are a large number of preferences stored outside the db.