This comment has been minimized.

After discussing with @zshipko it appears than we don't want to ship the OCaml graphQL client right now. They still need a few more tweaks before being totally happy with it. So it's better to not ship something that we know will change for sure in the very near future.

This comment has been minimized.

edited

I think #659 is what we discussed in Marrakesh which should work (NB: I haven't tested this, Sys.getcwd () returns an absolute path, now . is a relative one -- is that ok?)

the third issue is that Canopy doesn't hold their store anymore (you should be able to reproduce by testing Engil/Canopy#95 and see whether it comes up with a webserver, for me it fails to read .config/uuid from the git remote (which is there according to printfs after the clone)):

This store () leads to an empty repository most of the time. I've worked around this issue by introducing a mutable cell that keeps a handle to the store, and is updated after each pull (see hannesm/Canopy@104546c for the commit) -- I'm not sure whether this is a bug in irmin or not, maybe the semantics should be as is!?

This comment has been minimized.

edited

Based on the outstanding PRs and issues it seems do-able to plan the release for April 3rd. This gives us two weeks to get everything together, which should be enough for what's listed below.

If anyone thinks that timeline is too aggressive then there's no harm in extending it, however I think it would be best to make the release as soon as possible. Also, let me know if there's anything I'm missing here.

Timeline

March 29th

finish outstanding PRs

finalize CHANGES.md and migration guide

make sure tutorial is completely up-to-date

April 1st/2nd

final code review

April 3rd

publish Irmin 2.0.0 to opam

After this we can start on the 2.0.x release, which will include a blog-post/larger announcement.

This comment has been minimized.

@zshipko I tested this on solo5-hvt, not sure whether the unix backend will take another path in dependencies. I will re-try this weekend in a fresh switch and record the steps. Thanks for attempting to reproduce.

This comment has been minimized.

@dinosaure@zshipko did you get more understanding of the root cause of the MirageOS issue? Should we delay the release? (it's perfectly fine to delay the release, but I just would like to understand what happens :p)

This comment has been minimized.

I'm currently on some issues of ocaml-git. At this stage, I'm currently on the weird bug about hvt, in other side, I'm waiting a feedback from @hannesm about a deep change on the smart protocol where I mostly fix it blindfolded. If you can post-pone the release about one day, I can give you a report tomorrow.

This comment has been minimized.

for MirageOS, it would be nice to add support of a kv_ro and kv_rw to the mirage command line utility which uses irmin+remote git as storage (see generic_kv_rw/ro). at configure-time I'd like to select which kv implementation is used (unix/mem/git), and runtime provide the necessary data (file path, maximum size, remote url) as boot parameters. NB: @linse and me tried and failed, and atm hardcoding irmin-kv/rw into unikernels :/ (see https://github.com/hannesm/functoria-playground for some failed attempts at dealing with functoria)

This comment has been minimized.

edited

in respect to irmin-mirage and git-mirage and the new release, I'd be really happy if push and pull would work on kvm/hvt/freestanding, as well as Canopy.

Canopy

Applying your patch does clone the store, but doesn't keep the data in the store () below (i.e. when pull is called (by main), and i dump the values after the Sync.pull_exn t upstream `Set, i see data -- but subsequent operations that use store () get an empty store -- canopy then complains on startup that it can't find .config/uuid)).

this is a unikernel that uses the mirage-kv API and irmin+git in the backend. It listens on tcp port 1234 and 1235.

if a client connects on port 1234, it pulls (Store.connect .. KV.fold) the given remote and prints it to the client.

if a client connects on port 1235 and enters a "key:value", this key -> value is pushed to the remote repository

issues i saw today:

pull works via https and the git protocol

push doesn't work

good news:

git push works with the git protocol (tested locally against a git_daemon 2.19.1) if i use mirage/ocaml-git#350 (which needs some more released packages (encore, ke (I think)) and changes here in #668 (needed by git master)) (in addition @dinosaure checkseum should have a release with the "cross-compilation" fixes which are already merged to master)

This comment has been minimized.

@samoht the current KV_RW interface is fine if you're the only writer - there's explicit push control (batch / write), but no control over pull at all.

the pull could either be periodically (not so nice) or a KV_RW should offer a specific sync operation, called by the application once it receives a webhook / notification that the backend changed. there's also need to control merge in the multiple writer store, but only during set and remove (these operations atm offer only a write_error, but no way to merge that).

This comment has been minimized.

After being away for the last week I'm trying to understand what is needed to make this release happen.

Is mirage/ocaml-git#353 what needs attention right now? I am happy to spend some time trying to understand what's happening there and can also take care of the pretty-printers and documentation update requested by @hannesmabove.

It looks like this means we will need to postpone the irmin 2.0.0 until after the next ocaml-git update since these latest changes depend on the development version of ocaml-git (see: #668). This is unfortunate, but if Canopy, and other stable applications are broken then it makes no sense to move forward until these things are dealt with.

This comment has been minimized.

so, is there any release plan? if nobody apart from myself is interested in irmin + git remote on MirageOS, then you should go ahead and release without supporting that. I'm rather low on time atm (including the foreseeable future), and thus won't be able to debug and test much further (I hope the above provided examples are helpful). What occured to me is that irmin-mirage now depends on graphql, and I'm not sure why this should be the case -- maybe the irmin-mirage opam package should be split into multiple ones?

This comment has been minimized.

Well we're waiting on a new ocaml-git release and @samoht is out of town for the next two weeks, so I plan on spending this time debugging any known issues/preparing and updating documentation.

At this point it's best to make a new issue for any problems you encounter (rather than adding it to this thread) - that way we can keep track of what needs attention between the 2.0.0 and 2.0.x releases.

I don't think we're really thinking about releasing a broken version of irmin-git + remote on MirageOS, however since I'm not specifically using it for anything it's hard to find and reproduce bugs. One things that could help with this is some kind of unikernel test-suite (I will need to think about this!)

As for the graphql dependency in irmin-mirage - this was added to both the unix and mirage backends in hopes to eventually replace the current HTTP server. I suppose that Irmin_mirage.Graphql could live in a package like irmin-mirage-graphql or something, is this an issue because of executable size or something else?

This comment has been minimized.

@zshipko sounds good to me. yes, the graphql is not only a build-dependency (which adds to build time), but also runtime code size for functionality i do not plan to use. I like to have systems without any http (server nor client) code in them ;)

unikernel test suite: well, i linked to smaller ones i developed just for irmin, there's also some work on an e2e mirage test system (see https://github.com/mato/e2e-mirage-solo5/), which may be useful to have with irmin unikernels (where you may need a git daemon elsewhere as well).

I commented since I saw that the release plan was early April, now it's rather end of April. I have some irmin+git unikernels deployed, thanks to @dinosaure there are fewer issues in git now, and fewer pins needed, but i guess e.g. the canopy issue should be looked into more deeply (you tried canopy on unix and it worked, without unix it didn't -- this would worry me if I'd be in the process of releasing a library which behaviour should be the same independent of the platform).

This comment has been minimized.

Yeah, sorry @hannesm! That is definitely another issue that should be looked at before the release. I haven't had any further luck in tracking down the bug though, so another set of eyes could be helpful here.