[htdig] db.words.db size

Hello,
if db.words.db hits 2GB in size, rundig stops working and the database
gets unusable (no hits at all).
It is not a filesystem issue, since my filesystem can handle file sizes
above 2GB.
I have ht://Dig 3.2.0b6 running.
Could the problem be bdb related?
Thanks,
Rainer

Thread view

Hello,
if db.words.db hits 2GB in size, rundig stops working and the database
gets unusable (no hits at all).
It is not a filesystem issue, since my filesystem can handle file sizes
above 2GB.
I have ht://Dig 3.2.0b6 running.
Could the problem be bdb related?
Thanks,
Rainer

Rainer Sokoll schrieb:
> Hello,
>
> if db.words.db hits 2GB in size, rundig stops working and the database
> gets unusable (no hits at all).
> It is not a filesystem issue, since my filesystem can handle file sizes
> above 2GB.
> I have ht://Dig 3.2.0b6 running.
> Could the problem be bdb related?
>
> Thanks,
> Rainer
>
Hi Rainer!
We had this problem about 2 years ago with ht://Dig 3.1.6. I don't know
if it is an issue with 3.2.0b6 because we're still sticking with 3.1.6.
ht://Dig has to be build with Large File Support. Look in the archive
for '2GB Problem with htdig'. It's a thread back from April 2004.
You will have to build ht://Dig with '-D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64'. With 3.1.6 we then hit the next limit if our
words.db reaches 4 GB (happened here when htdig got into a loop because
of a misconfigured apache). This is the limit for Berlkey DB version 2.
I think 3.2.0b6 uses a newer version so this shouldn't trouble you.
Regards
Berthold
--
Dipl. Chem. Dr. Berthold Cogel University of Cologne
E-Mail: cogel@... ZAIK-US (RRZK)
Tel.: +49(0)221/470-7873 Robert-Koch-Str. 10
FAX: +49(0)221/478-85845 D-50931 Cologne - Germany

On Fri, Jan 26, 2007 at 05:06:38PM +0100, Berthold Cogel wrote:
> Rainer Sokoll schrieb:
> >Hello,
> >
> >if db.words.db hits 2GB in size, rundig stops working and the database
> >gets unusable (no hits at all).
> >It is not a filesystem issue, since my filesystem can handle file sizes
> >above 2GB.
> >I have ht://Dig 3.2.0b6 running.
> >Could the problem be bdb related?
> You will have to build ht://Dig with '-D_LARGEFILE_SOURCE
Large file support has been compiled in, but not
> -D_FILE_OFFSET_BITS=64'.
I'll give it a shot and report.
> With 3.1.6 we then hit the next limit if our
> words.db reaches 4 GB (happened here when htdig got into a loop because
> of a misconfigured apache). This is the limit for Berlkey DB version 2.
> I think 3.2.0b6 uses a newer version so this shouldn't trouble you.
It's linked against the latest version from sleepycat.
Thank you,
Rainer

On Fri, Jan 26, 2007 at 05:06:38PM +0100, Berthold Cogel wrote:
> >if db.words.db hits 2GB in size, rundig stops working and the database
> >gets unusable (no hits at all).
> You will have to build ht://Dig with '-D_LARGEFILE_SOURCE
> -D_FILE_OFFSET_BITS=64'. With 3.1.6 we then hit the next limit if our
> words.db reaches 4 GB (happened here when htdig got into a loop because
> of a misconfigured apache). This is the limit for Berlkey DB version 2.
> I think 3.2.0b6 uses a newer version so this shouldn't trouble you.
Yeah!
db.words.db now has 6.8 GB, and everything seems to be fine :-)
Thanks a lot!
Rainer
PS: I think this should go into the FAQ

Community

Help

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

CountryState

JavaScript is required for this form.

I agree to receive quotes, newsletters and other information from sourceforge.net and its partners regarding IT services and products. I understand that I can withdraw my consent at any time. Please refer to our Privacy Policy or Contact Us for more details