From ...
Path: archiver1.google.com!news1.google.com!newsfeed.stanford.edu!skynet.be!skynet.be!ossa.telenet-ops.be!nmaster.kpnqwest.net!nreader1.kpnqwest.net.POSTED!not-for-mail
Newsgroups: comp.lang.lisp
Subject: Re: CMUCL18d on Alpha?
References: <3CBA8472.48774F5C@ilt.fhg.de> <3CBBE1E6.93F85F44@ilt.fhg.de> <3CBBF4AC.39CD4A4C@kfunigraz.ac.at> <3CBE88D1.6A17BCE1@ilt.fhg.de> <4npu0xrqo5.fsf@rtp.ericsson.se> <3CBFBFFA.AF340500@ilt.fhg.de> <87bscfeo6l.fsf@orion.bln.pmsf.de>
Mail-Copies-To: never
From: Erik Naggum
Message-ID: <3228225108884071@naggum.net>
Organization: Naggum Software, Oslo, Norway
Lines: 43
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.2
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Date: Fri, 19 Apr 2002 17:11:49 GMT
X-Complaints-To: newsmaster@KPNQwest.no
X-Trace: nreader1.kpnqwest.net 1019236309 193.71.199.50 (Fri, 19 Apr 2002 19:11:49 MET DST)
NNTP-Posting-Date: Fri, 19 Apr 2002 19:11:49 MET DST
Xref: archiver1.google.com comp.lang.lisp:32026
* "Pierre R. Mai"
| Actually, since you've got millions of floats, I think that they
| should be read directly into a specialized vector, otherwise the list
| and boxing overhead is going to be huge, and might be the reason you
| are running into the 18b heap limit. On the x86 using a fairly
| current CMUCL (post 18c, but pre 18d), I can read 2 million random
| single floats (3 per line, making a 20MB file) in around 16-18MB of
| heap space, doubles should be double that amount (not tested), with:
|
| (defun read-floats-slow (stream)
| (do ((result (make-array 1000 :element-type 'single-float :adjustable t
| :fill-pointer 0))
| (float (read stream nil stream) (read stream nil stream)))
| ((eq stream float)
| result)
| (vector-push-extend float result)))
|
| Reading takes 80s for the file on an AMD K6-2/550.
If you start off with a vector that is pretty close in size to what you
expect, you will also avoid all the overhead of copying the vector you
have just extended several times. Depending on the implementation, you
may either extend the vector a million times (Allegro CL, with a default
extension of 20) or about 15-20 times (CMUCL doubles the vector's size),
or something entirely different; neither CLISP nor LWL provide reasonably
readable disassembly to understand what they are doing. In any case, if
you have any means of waiting until you know how many floats to allocate
space for, you will be better off. One way to do this is actually to
allocate, fill, and collect some fairly large "chunks" of the total, like
1024-element vectors. At the end of the read phase, you know how much
space you need, and you can write a specialized accessor that uses the
1024-element vectors directly in a two-level vector (note the simple
10-bit shift instead of an expensive division), or you can copy them all
into one vector of known size. This _should_ have a significant effect
on the time it takes to read all the data, but my guess is that it will
all pale in comparison to what you wil do with all this stuff in memory.
///
--
In a fight against something, the fight has value, victory has none.
In a fight for something, the fight is a loss, victory merely relief.
Post with compassion: http://home.chello.no/~xyzzy/kitten.jpg