However you should never be removing files manually from the Calibre library as then you will end up with a mismatch between the file system and the metadata database. Books should always be removed from Calibre using Calibre itself.

There a file called metadata.db at the root of the Calibre Library folder. It hold the full metadata for each book. Individual books can hold the sub-set of metadata that is supported by a particular book format. The metadata database has pointers to the files stored within the Calibre library. If you delete book files manually then you end up with orphan entries in the Calibre database.

If you have duplicates then both copies will show up from the Calibre GUI. Removing books via the Calibre GUI will delete the files, but will also remove the relevant entries from the metadata database.

If you have already deleted book files, then I think if you use the option to run the Calibre database integrity check (from Preferences->Advanced) it might correct these orphan entries.