Basically, the C library considers a keystone to be so large and heavy, they are kept on disk and mapped in, to ensure that that only the section of its data you actually access resides in memory. (While I would agree that very large keystones are possible – larger than the RAM in most PCs – I think this design is over-optimizing for the worst 1-2% of cases.)

Compared to idiomatic Rust iterators – with all of their wonderful multi-threading properties you get for free – this C “iterator API” is very clunky. And I’m trying to decide what API should be on my Rust wrapper object, and how much clunkiness it should reflect.

The options I’m currently considering are (feel free to suggest others):

Write an iter_keystone method the way Rust usually does iterators, and simply have it hold a mutable reference to the object instead of a shared one. This seems counterintuitive compared to expectations on iterators an their function names, and highly discouraged.

Write an iter_keystone_mut function the way Rust usually does iterators, and simply not have an iter_keystone function. There would be one comment buried in the rustdocs explaining why, but it just feels… weird to do that.

Write a standard mutable iterator API like option 2, but name the method something different, like iterate_keystones_from_disk. This would be a hint it has complex internals… but is that hint really necessary, and worth typing that long ugly string?

Expose the C API very directly, with methods keystone_iter_init and keystone_iter_next on my ComplexStructure wrapper type, adding some affordances like making next return an Option<KeystoneWrapper> instead of having a separate accessor function. But this just seems to be pushing the clunkiness off on the library’s users, which seems impolite.

Completely abandon the iterator idea, and have a read_keystone(index) method be the only way to read them at all. But Rust iterators just so useful, this would really be another way of pushing the problem off onto the library’s users – the problem this time being making an iterator out of an accessor function.

Create Arc-and-Mutex-based parallelism over the C API, where every access to a keystone (both iterators and object methods) locks a member of the wrapper object, moves the C iterator to the offset it wants, deep copies the data, and then releases it. This is the most effort for me, the most bug prone, and ignores the “heaviness” of keystones, but would give Rust users the most intuitive API.

but it’s a bit dodgy that structure->keystone_index exists and is mutable. Can you duplicate it for each iterator? If you can’t then having only iter_mut() is still viable and safe.

You can’t however implement Iterator directly on the ComplexStructure, since borrow checker won’t allow a structure to iterate and mutate itself at the same time (let a = s.next(); let b = s.next(); (a,b) must be valid).

It’s fine if you implement the usual C API methods on the ComplexStructure wrapper. You can also implement many traits such as Clone, Debug, or AsRef. The exception is the Iterator trait, which in practice needs to be on a separate object, in order to have its iteration state mutated separately from the object it iterates.

You can’t implement Rust methods directly on the CStructure type, as it’d imply it’s a Rust native struct that can be allocated, copied and moved in Rust, but it can’t, since C would cause crash if it called free() on Rust’s pointer.

Note the deep copy in there, because the region in C the pointer points to could be reused out-from-under Rust.

That’s the deep copy C tried to avoid, relying on mmap instead. And I suspect putting that into the Rust wrapper would be a little unexpected – in addition to having to make every other access by every other method lock a mutex.