My calibre metadata file somehow became corrupted and now my library is all screwed up according to calibre, but all the books are actually there. For most of the books it thinks there isn't an associated file.

Is there a way to fix this, other than rebuilding it from scratch? That would make me want to cry because I'm using 3 readers at the moment and I would lose the data about which book is on each reader.

My calibre metadata file somehow became corrupted and now my library is all screwed up according to calibre, but all the books are actually there. For most of the books it thinks there isn't an associated file.

Is there a way to fix this, other than rebuilding it from scratch? That would make me want to cry because I'm using 3 readers at the moment and I would lose the data about which book is on each reader.

I believe the easiest way to correct this would be to replace the metadata.db file with a recent backup.

If it helps at all, every time I tell calibre to "open containing folder" a new empty folder appears with a new number.

For example I have a folder

Calibre Library > Chris Nicholson > The Elephant Keeper (1299)

but if I select The Elephant Keeper and choose "open location" a new folder is created:

Calibre Library > Chris Nicholson > The Elephant Keeper (1892)

Is there a way I can link my books back properly?

So, I guess what I'm asking is, is there a way for me to edit the metadata.db file to give all the books their proper numbers. I'd rather do that for 1700+ books that have to reimport them and re-edit all their metadata by hand.

If it helps at all, every time I tell calibre to "open containing folder" a new empty folder appears with a new number.

It's not being created when you "open containing folder," it's opening the folder that Calibre thinks holds your book.

Quote:

For example I have a folder

Calibre Library > Chris Nicholson > The Elephant Keeper (1299)

This folder was created by Calibre previously.

Quote:

but if I select The Elephant Keeper and choose "open location" a new folder is created:

Calibre Library > Chris Nicholson > The Elephant Keeper (1892)

This is where Calibre thinks the book is located. It's almost certainly not created when you choose "open location" as that code doesn't create folders.

Quote:

Is there a way I can link my books back properly?

So, I guess what I'm asking is, is there a way for me to edit the metadata.db file to give all the books their proper numbers. I'd rather do that for 1700+ books that have to reimport them and re-edit all their metadata by hand.

I'd suggest this.
First, confirm what I wrote above (that the empty 1892 folder already exists.) I'm 95% certain, but doublecheck.
Next, check to see if you have the 1299 entry in your GUI main screen. I'm pretty sure from your post that you don't, but just make sure by doing a search or manually looking through to make sure you do not have these records at the bottom of your listing.
Third, I'm assuming that you have good metadata in the empty 1892 record, but no formats in that record. Also, assuming you do not still have the old entries (checked in step 2), open the 1299 folder (with a file explorer outside Calibre) and drag the book format files in it into Calibre's GUI.
This will create a third book id (lets say 1300), but without metadata.
Last, select both of 1299 (good metadata, no book formats) and 1300 (book formats, but no metadata) and choose the Edit Metadata, Merge-delete others to merge the two and keep your good metadata. (Preferably, select 1299 first, but it doesn't matter a lot.)

If you want, you can simply add the entire contents of your old library in one shot, then merge up all the records, or try to set Calibre to do this automatically for you (see below).

There are lots of different ways to do this that may make it easier. It may be best to move your library first using Calibre's new move library function. That will move what Calibre thinks are the correct folders and separate them from all the old folders (that Calibre seems to have forgotten about). Then, if you set set the regex in Add/Save correctly, and turn on the "If similar author title found ..." option, Calibre will be able to automatically match each new book to the correct old record (with good metadata) and you'll avoid having to do all the manual merges. Adding the old books should be trivial if you get them separated. It will take some testing to get it right, so if I was in your shoes, I'd back it all up and try several approaches to fix it, until I found one that would work well.
Good luck!

Next, check to see if you have the 1299 entry in your GUI main screen. I'm pretty sure from your post that you don't, but just make sure by doing a search or manually looking through to make sure you do not have these records at the bottom of your listing.

The numbers are the row number assigned to the book when calibre imports it into the database. It's nothing I can search on. These are in the metadata.db file, not in the individual book file. They are used as a locator, not as part of the book's individual metadata.

Quote:

Originally Posted by Starson17

Third, I'm assuming that you have good metadata in the empty 1892 record, but no formats in that record. Also, assuming you do not still have the old entries (checked in step 2), open the 1299 folder (with a file explorer outside Calibre) and drag the book format files in it into Calibre's GUI.
This will create a third book id (lets say 1300), but without metadata.
Last, select both of 1299 (good metadata, no book formats) and 1300 (book formats, but no metadata) and choose the Edit Metadata, Merge-delete others to merge the two and keep your good metadata. (Preferably, select 1299 first, but it doesn't matter a lot.)

This all makes no sense, because the problem lies in the metadata.db file not in the book metadata. I can't even select the actual book because calibre doesn't know it exists, as it isn't properly coded in the database. calibre is literally blind to that book because it has improper directions on how to reach it.

Quote:

Originally Posted by Starson17

There are lots of different ways to do this that may make it easier. It may be best to move your library first using Calibre's new move library function. That will move what Calibre thinks are the correct folders and separate them from all the old folders (that Calibre seems to have forgotten about). Then, if you set set the regex in Add/Save correctly, and turn on the "If similar author title found ..." option, Calibre will be able to automatically match each new book to the correct old record (with good metadata) and you'll avoid having to do all the manual merges. Adding the old books should be trivial if you get them separated. It will take some testing to get it right, so if I was in your shoes, I'd back it all up and try several approaches to fix it, until I found one that would work well.
Good luck!

And none of this will work because calilbre doesn't know that any of the correct folder exist. I need to fix the "map" so calibre is getting the correct "directions."

The folder is not there until I choose "open location." This was the first thing I checked.

Well, that's why I said to check it. Still, it's very odd.

Quote:

And calibre certainly does create folders. When you import a new book it creates a folder for that author if not already in the database and a folder for the book.

Of course it creates folders when you add, but when you "open location" it's trying to open a folder that it created when the book was first added. It should never have to create a folder at that point. Could you have changed drive letters? The folder it's "creating" is already in the Calibre database, so I assume it's trying to open a non-existent location and part of that process creates it if it doesn't exist. in normal operation, that should never happen.

Quote:

The numbers are the row number assigned to the book when calibre imports it into the database.

Third, I'm assuming that you have good metadata in the empty 1892 record, but no formats in that record. Also, assuming you do not still have the old entries (checked in step 2), open the 1299 folder (with a file explorer outside Calibre) and drag the book format files in it into Calibre's GUI.
This will create a third book id (lets say 1300), but without metadata.
Last, select both of 1299 (good metadata, no book formats) and 1300 (book formats, but no metadata) and choose the Edit Metadata, Merge-delete others to merge the two and keep your good metadata. (Preferably, select 1299 first, but it doesn't matter a lot.)

This all makes no sense, because the problem lies in the metadata.db file not in the book metadata. I can't even select the actual book because calibre doesn't know it exists, as it isn't properly coded in the database. calibre is literally blind to that book because it has improper directions on how to reach it.

I'm not sure what you don't understand here. I'm telling you to drag a book file from a file explorer (outside Calibre) into Calibre's GUI. That causes Calibre to create a new record in metadata.db. (It will also generate a new copy of the book.) That new record now points to the new copy of the book format. It does not have a copy of your old metadata. Then I'm telling you to merge the two records. The reason for the previous checks was to make sure that you didn't already have the two records you need to merge. If you did, you could have skipped the drag-to GUI step. Since you didn't, you needed to create a record that points to a book so you can merge the two.

Quote:

And none of this will work because calilbre doesn't know that any of the correct folder exist. I need to fix the "map" so calibre is getting the correct "directions."

The purpose of dragging the file from the library (which Calibre does not know about) into Calibre is to create a record that points to the book.

If it helps any, I wrote the merge record code. I also wrote the code that handles "If books with similar authors and titles found, merge automatically" so I'm intimately familiar with the database structure of Calibre.

I walked you through the simple solution of getting two records into Calibre that you could use my merge code on - one with metadata, and one with pointers to the actual book formats. I started there so you could see one option, however, that manual method is slow. The automatic merge will do the job much faster.

It is possible to "fix the map" as you say, but it's harder than the automatic method described.

Starson 17, thanks for pointing out the merge function. It's making the readding my 1700+ books easier. I didn't know about that.

I add the new book and merge it with the incorrect one that has the correct metadata. It's still 1700 hand-added entries, but still an improvement.

-Marcy

I'm glad it's helping. That's why I wrote it. Still, the automatic merge might make it much faster. I assume you are adding from your original copy of the book? Otherwise, you're doing what I told you to do in the step you didn't understand - adding the copy that's in the folder Calibre has forgotten about. The reason I told you to use the book in that folder, and not your original, is because Calibre will have stored it with exactly the same author/title in the filename as used in your remaining metadata record. That means you will always have a perfect match should you try to use "If book with similar author title found" to do the whole job automatically.

BTW, when you get done, you should run the Check database integrity and or move the database. Otherwise, you will have lots of junk folders mixed into Calibre's library folder. I assume you will backup before doing that.

Even if I add the copy that calibre has forgotten about I have to hand edit some entries. Tags that came with books that I delete pop back up. I don't know why. I don't want to automatic merge because I can miss books that way. It's tedious but I want to make sure every book is replaced.

One issue I have with the merge is that it won't delete tags from later selected books. I assume it was designed that way, but that's a big part of what I'm trying to accomplish.

A way to change the M shortcut to keep all the books would be helpful. That seems like it would be the most common use—to merge books in different formats without losing any of the books.

Even if I add the copy that calibre has forgotten about I have to hand edit some entries.

Tags that came with books that I delete pop back up. I don't know why. I don't want to automatic merge because I can miss books that way. It's tedious but I want to make sure every book is replaced.

If you're hand editing, it sounds like you want to do more than just repoint the old entries. You may find the following search to be useful:

Code:

formats:false

It will show you all records that don't have any actual books in them. That's how I'd make sure that I missed no records if I was doing it automatically. I assume that you had at least one book format in each record before you had your problem.

Quote:

One issue I have with the merge is that it won't delete tags from later selected books. I assume it was designed that way, but that's a big part of what I'm trying to accomplish.

Merge is designed to keep all the metadata it can, without losing any, so yes, if you have a tag in any of the merged records, it's kept. If you don't want certain tags, you can always delete them.

Quote:

A way to change the M shortcut to keep all the books would be helpful. That seems like it would be the most common use—to merge books in different formats without losing any of the books.

You should have only one record for each book, with all the formats for that book in that single record. That's what the "merge with delete others" does, it produces that single record, keeps all metadata in all the records, and gets rid of duplicate formats and extra records.

That's the most common use, by far. In fact, I almost didn't include the "merge-keep others" option, as that wasn't even in my mind when I started to write the code.

The "merge-keep others" option is used far less often. It's mostly used to make copies of other records for testing, or where you want to keep two different versions of the book in the same format (I have a collection of two column pdf classics and another collection of single column pdf classics that I keep.)

If you're hand editing, it sounds like you want to do more than just repoint the old entries. You may find the following search to be useful:

Code:

formats:false

It will show you all records that don't have any actual books in them. That's how I'd make sure that I missed no records if I was doing it automatically. I assume that you had at least one book format in each record before you had your problem.

I was doing the same thing by sorting on file size. The entries with no books attached have no file size.

The problem is that the whole calibre folder is wonky with some books missing. I have a backup folder with those books but it is missing other books, so have to switch between the two. Plus I need to make sure the right record is the dominant one in the merge process. One of the most important items I need to keep is the date. I use it to keep track of my reading. The re-added files often add with today's date and I don't want to lose the old date.