I would like to ask whether it is possible to pass parameters to functions by reference?

As far as I understand, function call parameters (even vectors, sequences, etc.) are passed by value (the form is evaluated, yielding the vector, sequence, etc. and the value is passed as a function parameter).

It seems to me, that by passing vectors and sequences by reference, it would be possible to reduce cons'ing and memory usage.

What are the general approaches to reducing memory usage by Lisp applications?

vityok wrote:As far as I understand, function call parameters (even vectors, sequences, etc.) are passed by value (the form is evaluated, yielding the vector, sequence, etc. and the value is passed as a function parameter).

I'm afraid you understand wrong (unless it's me). There is no implicit copying in Common Lisp. Simplifying (hopefully), there are two kinds of objects: immutable (numbers, characters), which are sort of passed by value, and everything else, which is passed by reference. One must remember that CL has no lists, only conses, which could lead to problems at some point, but vectors are fine.

On the other hand, places cannot be passed to functions, since they are not first class objects. But I am not sure if this has anything to do with your question.

The calculations are simple (like computing mean value over some samples) and, generally speaking, could be made in constant space:

one vector to hold the sum, and

one vector to hold the read sample

some helper variables

However, processing a 515K GZIP archive with approx 4600 lines of CSV records cons'es approx. 40M of memory. This is not yet critical as the program was already run with 30M GZIP files, but there is a 300M file on the way and I would like to reduce memory consumption as much as it is reasonable before processing this archive.

vityok wrote:I would like to reduce memory consumption as much as it is reasonable before processing this archive.

Well, it is hard to say anything specific without looking at the code, but note that with generational garbage collection allocating and the throwing away object is pretty fast, so usually you only need to care about things which remain allocated thorough the run. Obviously, decompressing a 300 MB file in memory might not be a good idea, but if it is streamed, I don't see how actual memory footprint would depend on size of the file.

vityok wrote:I would like to ask whether it is possible to pass parameters to functions by reference?

As far as I understand, function call parameters (even vectors, sequences, etc.) are passed by value (the form is evaluated, yielding the vector, sequence, etc. and the value is passed as a function parameter).

It seems to me, that by passing vectors and sequences by reference, it would be possible to reduce cons'ing and memory usage.

The best way to think about this is that all Lisp objects are allocated in the heap when they are created (even integers and characters), and they are always referenced by a pointer. The pointer is always passed to each function, so there isn't any copying involved (except for perhaps the pointer onto a stack, etc.). Now, putting everything into the heap creates garbage needlessly and is inefficient for some small data types (notably integers). Thus, most implementations actually fuse the small data types with the pointer for the sake of efficiency. A few of the low-order bits (called tag bits) are typically used to determine whether the pointer value is really a pointer value to something on the heap or a small immediate value. When Ramarren says that there are two types of objects, that's what he's referring to.

Now, because objects are always passed by reference, if you're working with a larger, aggregate object (list, vector, CLOS object, etc.), you can actually modify that larger object using mutable functions. Thus, it's quite common to pass vectors into functions where the data values in the vector are written with new values. With lists, because there is a lot of underlying structure there (linked cons cells), you have to be a little more careful about modifying them than with vectors. If you change the head of the list for any reason, the reference that the calling function had passed in is no longer valid and so you typically have to return a reference to the new list. Look at NCONC, for instance. It destructively joins the two lists, but it returns a reference to the new list.

What are the general approaches to reducing memory usage by Lisp applications?

Generally, you can reduce memory consumption in similar ways as you would any GC'd language such as Java--by reusing data structures and destructively modifying them. You'll find a lot of duality in the Lisp library, where there are both pure functional library functions that create new lists (e.g. APPEND) as well as a destructive versions of the same operation (e.g. NCONC). As Ramarren said, though, you'd be better off not succumbing to premature optimization. Garbage does take time to collect, but depending on what your program is doing, that may or may not be very significant. Don't try to program Lisp like you would C, for instance. You'll end up creating a lot of bugs for yourself (accidentally modify something that you shouldn't) and it might not bring as much benefit as you'd like.

The SBCL implementation of Common Lisp has a good compiler, a good optimizer, and a generational GC. I would write your implementation in the most natural style and then use the profiler to figure out where the slow parts are after you have determined that the natural implementation style doesn't meet your performance goals.

vityok wrote:With lists, because there is a lot of underlying structure there (linked cons cells), you have to be a little more careful about modifying them than with vectors. If you change the head of the list for any reason, the reference that the calling function had passed in is no longer valid and so you typically have to return a reference to the new list. Look at NCONC, for instance. It destructively joins the two lists, but it returns a reference to the new list.

Not to be too pedantic, for it is, in general, good style to make sure that the result of a destructive function is preserved, e.g., saving the results of SORT, or DELETE, whose output may not be eq to their input. From the HyperSpec entry on NCONC, however,

Just a quick note: arguments are always passed by value, i.e. copied, in Common Lisp, but here "arguments" denotes "implicit pointers to objects" (disregarding optimization tricks). That is why the function below will never change whatever object it is called with.

death wrote:Just a quick note: arguments are always passed by value, i.e. copied, in Common Lisp, but here "arguments" denotes "implicit pointers to objects" (disregarding optimization tricks). That is why the function below will never change whatever object it is called with.

Yes, you're right. I said that wrong. The references are always copied, so it's call by value, but if the reference is for a whole list, the list itself is not copied. So, as you say in your example, x is a binding to a value. But if you did this:

tayloj wrote:Not to be too pedantic, for it is, in general, good style to make sure that the result of a destructive function is preserved, e.g., saving the results of SORT, or DELETE, whose output may not be eq to their input.

Yea, that was a bad example to grab out of thin air. I just reached for the first destructive function that came to mind. Something like NREVERSE would have been a better example, or SORT and DELETE as you mention.