Bulat Ziganshin wrote:
> Hello Arne,
>> Wednesday, November 19, 2008, 11:57:01 AM, you wrote:
>>>> finding that it uses about twice as much memory as I had anticipated.
>>>>Hello, and thank you for your reply.
> it may be
> 1) GC problem (due to GC haskell programs occupies 2-3x more memory
> than actually used)
>I wasn't aware of that - but it should be possible to trigger a GC after
loading a whole lot of data?
> 2) additional data (you not said how long each small array. you should
> expect 10-30 additional bytes used for every array)
>The arrays represent the netflix data set: 100 000 000 ratings, given
for 17770 films.
For each the films, I want to hold (on average, roughly) 2000 ratings,
held as one person id (32-bit) and one rating (8-bit), in the respctive
arrays.
(In addition, I want to be able to load the inversion of this data: for
all persons, I want to hold their ratings in a similar way:
16-bit film id, 8-bit rating. There are 480000 persons, so this should
be on average 200 entries per person.
I have coded a few approaches to inverting this, but I can't allocate
the array before traversing the data, because I don't know the sizes.
How can one go about inverting this data in memory?
It seems that any kind of laziness will fill the whole memory before I
have traversed the whole set - and if I use several accumArrays, it
seems that it will hold the whole uncompacted dataset in memory between
accumArrays.
Ideally I want to hold all ratings as well as statistics for all films,
and the same for all the persons - and then have room to spare for
running an algorithm...
Best regards,
Arne D Halvorsen