4.8 Memory management

According to the documentation for curl_easy_setopt, the type
of the third argument when option is CURLOPT_ERRORBUFFER
is char*. Above, we’ve defined
set-curl-option-errorbuffer to accept a :pointer as the
new option value. However, there is a CFFI type :string,
which translates Lisp strings to C strings when passed as arguments to
foreign function calls. Why not, then, use :string as the
CFFI type of the third argument? There are two reasons, both
related to the necessity of breaking abstraction described in
Breaking the abstraction.

The first reason also applies to CURLOPT_URL, which we will use
to illustrate the point. Assuming we have changed the type of the
third argument underlying set-curl-option-url to
:string, look at these two equivalent forms.

The latter, in fact, is mostly equivalent to what a foreign function
call’s macroexpansion actually does. As you can see, the Lisp string
"http://www.cliki.net/CFFI" is copied into a char array and
null-terminated; the pointer to beginning of this array, now a C
string, is passed as a CFFI:pointer to the foreign
function.

Unfortunately, the C abstraction has failed us, and we must break it.
While :string works well for many char* arguments, it
does not for cases like this. As the curl_easy_setopt
documentation explains, “The string must remain present until curl no
longer needs it, as it doesn’t copy the string.” The C string
created by with-foreign-string, however, only has dynamic
extent: it is “deallocated” when the body (above containing the
foreign-funcall form) exits.

If we are supposed to keep the C string around, but it goes away, what
happens when some libcurl function tries to access the
URL string? We have reentered the dreaded world of C
“undefined behavior”. In some Lisps, it will probably get a chunk
of the Lisp/C stack. You may segfault. You may get some random piece
of other data from the heap. Maybe, in a world where “dynamic
extent” is defined to be “infinite extent”, everything will turn
out fine. Regardless, results are likely to be almost universally
unpleasant.7

Returning to the current set-curl-option-url interface, here is
what we must do:

(let(easy-handle)(unwind-protect(with-foreign-string(url "http://www.cliki.net/CFFI")(setf easy-handle (curl-easy-init))(set-curl-option-url easy-handle url)#|do more with the easy-handle, like actually get the URL|#)(when easy-handle
(curl-easy-cleanup easy-handle))))

That is fine for the single string defined here, but for every string
option we want to pass, we have to surround the body of
with-foreign-string with another with-foreign-string
wrapper, or else do some extremely error-prone pointer manipulation
and size calculation in advance. We could alleviate some of the pain
with a recursively expanding macro, but this would not remove the need
to modify the block every time we want to add an option, anathema as
it is to a modular interface.

Before modifying the code to account for this case, consider the other
reason we can’t simply use :string as the foreign type. In C,
a char * is a char *, not necessarily a string. The
option CURLOPT_ERRORBUFFER accepts a char *, but does
not expect anything about the data there. However, it does expect
that some libcurl function we call later can write a C string
of up to 255 characters there. We, the callers of the function, are
expected to read the C string at a later time, exactly the opposite of
what :string implies.

With the semantics for an input string in mind — namely, that the
string should be kept around until we curl_easy_cleanup the
easy handle — we are ready to extend the Lisp interface:

(defvar*easy-handle-cstrings*(make-hash-table)"Hashtable of easy handles to lists of C strings that may be
safely freed after the handle is freed.")(defun make-easy-handle ()"Answer a new CURL easy interface handle, to which the lifetime
of C strings may be tied. See `add-curl-handle-cstring'."(let((easy-handle (curl-easy-init)))(setf (gethash easy-handle *easy-handle-cstrings*) '())
easy-handle))(defun free-easy-handle (handle)"Free CURL easy interface HANDLE and any C strings created to
be its options."(curl-easy-cleanup handle)(mapc #'foreign-string-free
(gethash handle *easy-handle-cstrings*))(remhash handle *easy-handle-cstrings*))(defun add-curl-handle-cstring (handle cstring)"Add CSTRING to be freed when HANDLE is, answering CSTRING."(car (push cstring (gethash handle *easy-handle-cstrings*))))

Here we have redefined the interface to create and free handles, to
associate a list of allocated C strings with each handle while it
exists. The strategy of using different function names to wrap around
simple foreign functions is more common than the solution implemented
earlier with curry-curl-option-setter, which was to modify the
function name’s function slot.8

Incidentally, the next step is to redefine
curry-curl-option-setter to allocate C strings for the
appropriate length of time, given a Lisp string as the
new-value argument:

(defun curry-curl-option-setter (function-name option-keyword)"Wrap the function named by FUNCTION-NAME with a version that
curries the second argument as OPTION-KEYWORD.
This function is intended for use in DEFINE-CURL-OPTION-SETTER."(setf (symbol-function function-name)(let((c-function (symbol-function function-name)))(lambda(easy-handle new-value)(funcall c-function easy-handle option-keyword
(if(stringp new-value)(add-curl-handle-cstring
easy-handle
(foreign-string-alloc new-value))
new-value))))))

A quick analysis of the code shows that you need only reevaluate the
curl-option enumeration definition to take advantage of these
new semantics. Now, for good measure, let’s reallocate the handle
with the new functions we just defined, and set its URL:

For fun, let’s inspect the Lisp value of the C string that was created
to hold "http://www.cliki.net/CFFI". By virtue of the implementation of
add-curl-handle-cstring, it should be accessible through the
hash table defined:

Looks like that worked, and libcurl now knows what
URL we want to retrieve.

Finally, we turn back to the :errorbuffer option mentioned at
the beginning of this section. Whereas the abstraction added to
support string inputs works fine for cases like CURLOPT_URL, it
hides the detail of keeping the C string; for :errorbuffer,
however, we need that C string.

In a moment, we’ll define something slightly cleaner, but for now,
remember that you can always hack around anything. We’re modifying
handle creation, so make sure you free the old handle before
redefining free-easy-handle.

Footnotes

“But I thought Lisp was supposed to protect
me from all that buggy C crap!” Before asking a question like that,
remember that you are a stranger in a foreign land, whose residents
have a completely different set of values.