Handling large volumes of data

Gary Jones

Greenhorn

Posts: 22

posted 13 years ago

Hi, I fire a query which fetches me about 1 million records in resultset. What approach is correct for handling and processing such a huge resultset. getting such a huge resultset object put a heavy load on the system jvm and may ocasionally result in OutOfMemory error.Is XML appropriate? Pl advise. Regards, GaryNYC

i guess that producing a plain-text-xml file is "better than" creating 1 million objects (but how should this xml be processed), but without more information what you want to do with 1 million rows in a single ResultSet it's difficult to give advice

AR

Gary Jones

Greenhorn

Posts: 22

posted 13 years ago

Of course there are turn arounds like pagination etc - but can we do it with a single hit to the database rather than reaching for the db everytime a new page is called.Even an xml stream with 1m rows will be big size.

Why do you need one million rows in memory? How big is a row? If a row is smaller than 100 bytes, you end with less than 100 MB. What do you want to do with that data? [ December 11, 2004: Message edited by: Stefan Wagner ]