Patches applied in the last week (52)

Thursday, November 19, 2009

The Darcs Hacking sprint was co-located in Vienna, Austria and Portland, Oregon, USA.
The Vienna team had a unique experience as the sprint happened to coincide with a student protest that resulted in an occupied University building. Great fun was had by all as we shared our hacking space with the Occupiers on Saturday.
The Portland team was graciously hosted by Galois. As a special treat, we got a visit from Thomas Hartman, one of the folks behind Patch-tag.

Overview

Among other things, the upcoming Darcs 2.4 release has some nice performance improvements from Petr Ročkai's hashed-storage work.
One first goal in this sprint was to polish up this work, making our hashed repository support good enough for GHC team to upgrade from the deprecated old-fashioned format. We think we've largely accomplished what we set out to do. Read on for more details!
Our second goal was to provide a space for new Darcs hackers to get started with making contributions to the project. We had 14 new Darcs developers at this sprint! We hope that they will stay on with the project and continue submitting patches.

Issues resolved

Note that some of this work is still pending patch review or amendements.

issue540 - darcs remove --recursive - Roman Plášil

issue835 - show files with arguments - Luca Molteni

issue1122 - get --complete UI - David Markvica

issue1208 - trackdown --bisect - Matthias Fischmann, Radoslav Dorcik

issue1224 - darcs convert on darcs-2 repositories - Tomáš Caithaml

issue1377 - hardcoding of darcs executable name - Benedikt Huber

issue1392 - use Parsec for authorspellings - Tomáš Caithaml

issue1394 - show time elapsed after each test - Kim Wallmark

issue1499 - versioned show files - Thomas Hartman

issue1500 - misleading darcs progress reports - Roman Plášil

issue1624 - break up global cache into subdirectories - Luca Molteni

issue1643 - optimize --upgrade should do optimize - Christian Berrer

Hacking

Studying darcs

Thomas DuBuisson and Jason Dagit studied some hanging merge cases on the GHC repository to see if we could garner some insights into them. While this did not yield much fruit, it did result a few code cleanups.
Jonathan Daugherty and Josh Hoyt took a tour of the Darcs source. We hope they'll be joining us as Darcs hackers in the future :-)
Jason Dagit produced a concrete and low-level description of the Darcs record command.

Code cleanups

As a warm-up exercise, everybody worked on tidying the Darcs and hashed-storage source, eliminating annoying GHC warnings and implementing suggestions from hlint.

Tomáš Caithaml simplified the darcs show authors authorspellings code by rewriting it in Parsec.
Jason Dagit cleaned up the elegant_merge function and removed commutex in favor of commute.
Jonathan Daugherty submitted some improvements to the user manual.

Performance improvements

Luca Molteni improved Darcs's handling of the global cache, splitting it into buckets for better performance.

Christian Berrer extended the darcs optimize --upgrade command to also act as plain old darcs optimize in addition to upgrading the repository format.
David Markvica modified darcs get --complete so that it no longer offers to create a lazy repository.
Benedikt Huber relaxed the assumption that darcs executable would be called "darcs", instead fetching this from the command line.

New features

Matthias Fischmann and Radoslav Dorcik worked on a new trackdown --bisect feature for darcs trackdown which makes trackdown more useful for larger sets sets of patches.
Roman Plášil implemented an oft-requested remove -r feature, making it easier to undo accidental adds of large directories.
Luca Molteni implemented darcs show files with an argument to limit the output to a specific file or directory.
Thomas Hartman implemented matchers support for darcs show files, allowing to get the list of files tracked by Darcs at a given point in the history (eg. darcs show files --tag 2.4).
Petr Ročkai worked on a new more extensible, self-documenting format mechanism as a future alterantive to _darcs/format.

UTF-8 Support

Reinier Lamers continued his work on storing darcs patch metadata in UTF-8. He worked on autodetecting UTF-8 content (in the patch metadata) and found some subtle bugs in the utf8-string library along the way.

SVN integration

The Vienna local team, Christian Berrer, Thomas Danecker and David Markvica, worked on SVN integration and studied the libsvn directory in great detail.

Zero Windows Bugs!

Salvatore Insalaco slayed the 18 Windows bugs that resulted from our recent hashed-storage work. This means that we can can definitely release the new hashed-storage work in Darcs 2.4.

Darcs Team infrastructure

Joachim Breitner made some improvements to the darcswatch core and user interface. He also added hooks to Darcswatch for integration with Darcs Team patch tracker. Joachim Breitner, Eric Kow, Petr Ročkai also added some polish to patch tracker, in particular to its email gateway interface and interaction with the darcs-users mailing list. The result of these efforts is a more efficient Darcs Team with simpler and smoother patch review process.
Kim Wallmark worked on making the 'cabal test' friendlier by outputting the amount of time elapsed after long tests.

Discussions

Priorities for the Sprint

Salvatore Insalaco, Eric Kow, Reinier Lamers, Luca Molteni and Petr Ročkai discussed the key goals for the sprint and also produced a list of the top ProbablyEasy.

Type witnesses

Jason Dagit gave a presentation about our type-witnesses, how the type hackery in darcs is implemented, and why it is useful.

Patch-tag

Thomas Hartman and the Portland crew discussed patch-tag: how it felt from a user perspective and what the most useful features would be.

SVN integration

Thomas Danecker, Eric Kow, Petr Ročkai, Ganesh Sittampalam (IRC) discussed the SVN integration project: what the roadmap should look like and how this feature might fit into Darcs, whether as a fully integrated feature, a plugin or a standalone application.

Petr Ročkai and Eric Kow also discussed the nature of patch/changeset based revision control (like Darcs) and snapshot based revision control (everything else). Petr observed that changeset based version control can be seen as just snapshot based control with a fixed diffing algorithm. To support things like SVN integration, Darcs will need to freeze its diffing algorithm.

Darcs roadmap

Salvatore Insalaco, Eric Kow, Reinier Lamers and all Vienna hackers developed the roadmap for Darcs 2.4 (January 2010) and Darcs 2.5 (July 2010). Using a chalkboard we listed the features we wanted, how desirable they were and what order we should do them in.

Windows support

Salvatore Insalaco, Eric Kow and Reinier Lamers discussed the future of Windows support in Darcs.
Official binary: Our first priority will be to get an official binary for Windows downloadable from Darcs.net. The upcoming Darcs release (Darcs 2.4 in January 2010) will the first to provide an official binaries. Currently, we have the resources to provide a binary for Windows only, but in the future the list of platforms may grow.
Refined integration: After the Darcs 2.4 release, we will work on developing a friendly Windows installer for Darcs which includes not just the Darcs binary but (optionally), TortoiseSSH for better ssh support and a tool for sending email via darcs send.
Cygwin support: We will not be able to support Cygwin, but we will work on providing more explicit documentation on what works (mostly everything) and what does not (absolute cygwin paths). We think that the improved Windows installer will provide a sufficiently comfortable Darcs experience for most Cygwin users.

Fun and speculation

Better Darcs Hacking Sprints

Every sprint teaches something new for what we hope to be a long tradition of biannual Darcs Hacking events.

One thing which worked out well this year was that we provided a clean list of the best ProbablyEasy bugs for new hackers to work on. This allowed new developers to make highly desirable contributions from the very beginning. This year the list was built with live discussion, a few Darcs Team members huddled in a hostel room over an open bug tracker. Perhaps we can replicate the success by having similar pre-sprint meetings in the future.
Replicating success is one thing; how can we do even better? Mainly we need to improve the experience that new hackers have. We were lucky this year to have many new Haskellers and Darcs developers, but we could have done a better job in helping them to get started.
First, some technical issues: A lot of our new developers lost the better part of Saturday morning setting up the machines to build Darcs and to send patches to the list. In the next sprint, we will be ready with much more precise and detailed setup instructions (developed hastily during this sprint) going all the way to the best default settings for Darcs, configuration files for msmtp (or similar sendmail replacements). It also has been suggested that we provide a virtual machine image for instant Darcs hacking.
Second, our mentoring strategy: We were fairly successful at providing in-depth individual mentoring; however what we could done better was to provide more mass mentoring at the very beginning, to help developers send their first patches. To make this work, we should try starting the next sprint with a group mentoring session with the specific purpose of getting people set up to send their first patches. We should encourage new developers to work in small teams, for example with an extra USB keyboard to facilitate pair programming on a single machine.

Attendees

Vienna

Joachim Breitner

Benedikt Huber

Eric Kow

Reinier Lamers

David Markvica

Petr Rockai

Radoslav Dorcik

Salvatore Insalaco

Matthias Fischmann

Thomas Danecker

Christian Berrer

Luca Molteni

Roman Plášil

Tomáš Caithaml

Pivo

Portland

Jason Dagit

Josh Hoyt

Thomas Hartman

Thomas DuBuisson

Kim Wallmark

Jonathan Daugherty

Thanks!

Many thanks to the local team at TU Vienna and Galois for a wonderful welcome! Thanks to the many generous donors who helped us to subsidise travel to the sprint. Thanks also to Microsoft Austria for sponsoring the sprint with drinks and snacks. Molte grazie to Salvatore Insalaco for a wonderful lunch on Sunday.
Finally, thanks to Petr and Luca for the photos, and more generally to everybody who participated in the sprint! It was great to have you and we hope to see you again either on the mailing list or at future sprints.

See you in March!

The fourth Darcs Hacking Sprint will be taking place in Zürich on 19-21 March 2010 as part of the Haskell Hackathon. Hope to see you there!

Wednesday, November 18, 2009

News and discussions

The third darcs hacking sprint happened last weekend in Vienna and Portland! An extensive report will be posted here soon. As a teaser of all the work done, here is a draft implementation of trackdown --bisect written in Vienna by Matthias Fischmann:

Patches applied in the last week (33)

Sunday, September 20, 2009

The darcs team would like to announce the immediate availability of a new stable version of darcs, 2.3.1. This is the latest development in the 2.3 branch, which brought significant performance improvements. Performance improvement continues to be a core focus of darcs development. Darcs 2.4, planned for January, will build upon the improvements in 2.3 to be even faster.

The user-visible changes in darcs 2.3.1 are:

Fix bugs in optimizations in 2.3.0

Documentation improvements

Remove autoconf build system

A more complete list of changes in this and older versions of darcs can be

The build instructions are available in the enclosed README file in this tarball. Moreover, if you have cabal-install available, you can install latest stable release of darcs by issuing the following commands (no tarballs needed):

$ cabal update

$ cabal install darcs

This should give you a darcs binary in `~/.cabal/bin` -- you should probably add this to your PATH. More detailed instructions for installing on Windows are available near the end of this announcement.

Installing on Windows

To install darcs on Windows systems from scratch, please download the Haskell Platform and MSYS:

After installing both, you should have an "MSYS" icon: run MSYS and in the terminal window type (the `$` character denotes the prompt, do not repeat it):

$ cabal update

$ cabal install darcs -f-curl

This should download, compile and install all required dependencies and also darcs itself. The resulting darcs executable will be placed into the Haskell Platform executables folder, and should be accessible from the MSYS shell (just type "darcs --version" to check).

3. Eric also met researchers working into Operational Transformation, and wrote a report of this exchange, aiming at making people of collaborative editing and revision control speak together and in a common language:

Patches applied in the last week (17)

Saturday, August 8, 2009

News and discussions

1. Following a proposal by Eric Kow, the darcs team switched to a commit-bit development model. Now there is no longer a Patch Manager, and several developers have commit rights to the main repository or darcs:

Issues resolved in the last week (0)

Patches applied in the last week (38)

Thursday, July 23, 2009

We are pleased to announce the release of Darcs 2.3.0. This is the second of our time-based (biannual) releases and we hope to have many more. Here are some of the key changes from the last release:

Better performance - we believe the darcs whatsnew command is now much faster for hashed repositories. If you notice these improvements, you may be pleased to know that Petr's Summer of Code project aims to extend these improvements to the rest of Darcs. In the meantime, you may also notice improvements in darcs record and darcs changes; reduced memory consumption in darcs check and repair; and also support for hardlinks on NTFS, which allows for faster repeated retrievals on Windows.

Easier installation - Darcs now uses the Cabal build method by default which greatly simplifies installation. Combined with the upcoming Haskell Platform, building Darcs from source is now painless, even under Windows. Future versions of Darcs will use Cabal only.

Improved repository integrity - The new "darcs gzcrcs" feature detects and repairs files with a broken CRC in your darcs repository and caches. To the best of our knowledge, these files were only produced by Darcs 2.0.2 - 2.1.2, and only within "old fashioned" (non-hashed) repositories, and only the CRCs are affected. In any case, we highly recommend upgrading to Darcs 2.3.0 so that you can repair any such files. See issue844 for more details.

Darcs 2.3.0 requires GHC 6.8 or higher, and we recommend using GHC 6.10 for best results. See the official release announcement for more details about this, as well as a more complete list of bug fixes and new features.

Patches applied in the last week (47)

Wednesday, May 13, 2009

We are pleased to announce that Darcs has joined the Software Freedom Conservancy. The Conservancy is an organisation composed of Free and Open Source Software projects.

Joining it gives us the benefits of a formal legal structure: the ability to collect donations and hold assets, as well as protection of Darcs hackers from legal liability. The Software Freedom Conservancy is a tax-exempt 501(c)(3) organization, so member projects can receive tax-deductible donations to the extent allowed by law.

Many thanks go to the SFC for their work to support the FOSS community.

Patches applied in the last week (47)

Friday, April 24, 2009

Our second darcs hacking sprint was held in Utrecht, the Netherlands from 17-19 April.

Overview

This is our second sprint, but it was also the first one that was held in a single location. Being all in one place was extremely helpful for us because it gave everybody a chance to interact with everybody else (especially with other Haskellers)! It also meant that instead of hacking a lot of code, we focused on sharing knowledge within the team and with the rest of the community, developing new ideas and planning for the future.

Total sprint patches : 24

Participants: 11 onsite and 3 offsite

Presentations

An alternative patch theory notation?

Marnix Klooster presented an alternative notation for patch theory based on priniciples elucidated by Edsger Dijkstra. The goal is to develop a way of writing patch theory that is easier to understand. For example, instead of treating patch commutation as a primitive, as in (p,q) ↔ (q',p'), we could talk about pushing a patch over another patch, (e.g. p ⊳ q), which also has the benefit of avoiding introducing new patch names such as p'. Also instead of writing the inverse of p as p-1, we could write them as -p.

question and answer session

On Saturday morning, Eric led a Question and Answer session in which the wider Haskell community could ask questions about darcs usage or share their experiences using darcs and learn more about the upcoming darcs 3 transition.

Some of the features we discussed are the patch matching --match features, and the notion of context files which give you a precise notion of repository version, and also the darcs annotate command.

One of the interesting experiences we learned from the Haskell community was that it is often useful to collect a set of completely unrelated repositories into a one. A problem that arises in this use case that repositories would have common filenames like "Makefile" that give rise to a conflicts. As a result of this discussion, Ganesh had some new ideas about how darcs could work to better support this scenario.

Thanks to Ben Moseley for the suggestion and to Ganesh and Petr for answers!

time-sliced HPC code coverage

Thorkil Naur gave a demonstration of his creative technique for using the HPC code coverage tool to examine why some darcs commands take so long. With the help of HPC and a small shell script, Thorkil can show what parts darcs code are being called at various time intervals during a long darcs session.

Discussions and implementation

The camp/darcs-3 transition

Ian, Ganesh and Eric discussed plans for transitioning to camp or darcs 3. We still have a lot of questions left to answer, but four points are clear to us:

We at least want the camp-core (patch theory 3) component to be part of future darcs.

Ian is focused on camp fundamentals for now, that is, proving things in Coq.

We anticipate at least three more darcs releases (2010-07) before this fundamentals work is done.

Meanwhile, the darcs team should work on converging to camp, for example, by turning as much code into small, reusable libraries as possible

More on the roadmap in a future e-mail!

Camp/darcs-3 and Coq

Ian and Florent (hacking from Berlin) worked on formal proofs of Camp theory using the Coq proof assistant. One of the major discussions that came up was how to merge their two approaches. As a result of this discussion, Florent has come up with a plan for merging their patch theory proof initiatives.

Overall, the sprint was very productive for Ian. He achieved his goal of finishing the proof that he was working on, and also did some refactoring of it, making it much simpler and shorter than it started off. This will be even more important as he gets to the larger proofs that are still to come. Ian also refactored everything to use the Coq module system properly. Being able to pick the brains of some more experienced Coq folk at the Hackathon was key to getting so much done. Also valuable were some good discussions, which have given him a lot to think about!

implementation

Proof completed and refactored by Ian

Some minor bugs fixed in the camp theory and paper.

Documentation

Daniel Carrera continued his work on making the darcs wiki and patch theory documentation friendlier.Despite a lot of recent technical difficulties -- the darcs team is in the middle of a switch from MoinMoin to a darcsit wiki -- Daniel has made some great progress and has put together a static demo of the new content.

Filecache optimisation

The filecache is a mapping from a filename at a certain point in history to the creation name of the file and all patches touching the file. It can be used to speed up commands such as "changes filename" or "annotate filename" where performance is important for repository browsing tools, e.g. darcsweb or darcsit.

Benedikt and other darcs hackers discussed the design of the filecache and how changes and annotate can be optimized by making use of it as well as the interaction between this and the new hashed-storage work . This lead to a simpler design of the filecache based on unique filenames (creation name + creation patch).

implementation:

Prototype that creates filecache with "optimize --filecache"and uses it for "changes filename" implemented. These will be submitted for review when "annotate" has been adapted.

Optimize parseDate by switching from String to ByteString to speed up reading the inventory file. Pending review.

Source file mechanism

Petr and Eric discussed the current problems with the caching mechanism used by hashed repositories.

Darcs's --hashed repositories make darcs a lot safer to use and also allow for performance gains such as --lazy patch fetching and the global patch cache. But there are some annoying bugs in its implementation, for example, stale entries in our cache could cause darcs to waste time connecting to and timing out from non-existent servers.

Petr has developed a plan to solve these issues by ignoring bad cache locations, warning about them, and deleting them when it is appropriate to do so. He has begun studying the darcs caching source code to prepare for future implementation work.

zlib and CRC errors

Ganesh also worked on the corrupted gzip patch files problem. After discussions with Duncan Coutts he implemented code in the Haskell zlib library to support continuing after finding patch corruption, and then added code to darcs to make use of this new API. Still to do is to clean up the zlib patches and submit to Duncan, and to make darcs repair able to fix the broken patches.

Test suite improvements

Reinier Lamers began some work on optimising the darcs QuickCheck test suite so that we could run it all the time. This turned out to be very challenging because the performance problems in the unit tests were less localized in two bugs than we expected beforehand. Reinier was able to fix one of these bugs, but the resulting speedup is not of an order-of-magnitude kind and is hardly noticable without a timer. Future work will be to further investigate how to reduce the number of patch equality tests, which will require us to delve further into the patch theory code.

implementation:

Profiling of darcs and test suites

Patches sent and applied for some small optimisations.

Future work: filenames and semi-conflicts

Spurred by a question from Peter Verswyvelen in the Darcs Q&A about merging two unrelated repositories into separate subdirectories, Ganesh figured out a way of making the resulting add-add conflicts be more manageable than in current darcs.

Roughly speaking, darcs could associate each filename with some sort of unique ID. If we then try to merge two unrelated repositories that (for instance), have the same filename, darcs need not treat them as an actual conflict. These files would have a semi-conflicted state, where darcs does not really think of them as being conflicting, but warns the user about the clashing filenames. This mechanism could also be easily extended to deal with issues such as case-insensitive filesystems, or specific unsupported filenames (e.g. COM1 on Windows).

Future work: conflict state

Nicolas Pouillard developed an proposal for storing conflict markings in a "conflict state" file, which can then be passed on to high level conflict resolution tools, for example with a graphical interface.

Future work: commute properties tool

Ganesh continued work on a tool he has been writing for some months which is intended to help with designing new patch types and their commutation rules. He doesn't have anything ready to unleash on the world yet, but hopes to soon.

Participants

In addition to the work listed above, there was lots of other interesting work and discussions, not to mention time invested in getting (re)acquainted with the darcs source (Arjan, Eric and Ben). Thanks to everybody that participated and hope to see in hacking sprint #3!

Here's the list of on-site participants in alphabetical order.

Arjan Boeijnk

Daniel Carrera

Marnix Klooster

Eric Kow

Ian Lynagh

Ben Moseley

Thorkil Naur

Nicolas Pouillard

Petr Ročkai

Benedikt Schmidt

Ganesh Sittampalam

Thanks to...

Haskell Hackathon organisers and sponsors

The Hac5 organisers did a brilliant job. We had Internet access, food and most importantly, a very warm welcome! Thanks to the Utrecht team for their very hard work getting the logistics down and congrulations for keeping 50-odd Haskell hackers happily hacking away.

Thanks also to the sponsors of the Hackathon for their generous support:

Darcs Donors

Speaking of generosity, this is the first darcs hacking sprint to be supported with donations from the darcs community. We raised $1072 with contributions from 24 donors. This allowed us to subsidize travel for three of our darcs hackers and have some money left over for future darcs hacking sprint.

Participants

Finally, thanks to all the participants of the Haskell Hackathon! It was a real joy for so many of us darcs hackers to be in a single room at the same time, and also for us to chat with folks from the wider Haskell community. Let's do this again!