ddc: Ticket #99: Finding which modules needs to be rebuilt is grossly inefficient.http://trac.haskell.org/ddc/ticket/99
<p>
We should also record its interaction with the server, so we can replay it during offline testing.
</p>
en-usddchttp://trac.haskell.org/ddc/chrome/common/trac_banner.pnghttp://trac.haskell.org/ddc/ticket/99
Trac 0.11.1erikdFri, 21 Aug 2009 22:12:34 GMTcomment addedhttp://trac.haskell.org/ddc/ticket/99#comment:1
http://trac.haskell.org/ddc/ticket/99#comment:1
<p>
Rover crashes on Linux x86_64 due to a stack overflow.
</p>
<blockquote>
<p>
&gt; bin/ddc test/90-Programs/Rover/Main.ds -o test/90-Programs/Rover/Main.bin
Stack space overflow: current size 8388608 bytes.
Use `+RTS -Ksize' to increase it.
</p>
</blockquote>
<p>
Doubling the stack size like this works:
</p>
<blockquote>
<p>
&gt; bin/ddc +RTS -K16M -RTS test/90-Programs/Rover/Main.ds -o test/90-Programs/Rover/Main.bin
</p>
</blockquote>
TicketerikdMon, 01 Feb 2010 04:56:20 GMTcomment added; owner set; status changedhttp://trac.haskell.org/ddc/ticket/99#comment:2
http://trac.haskell.org/ddc/ticket/99#comment:2
<ul>
<li><strong>owner</strong>
set to <em>erikd</em>
</li>
<li><strong>status</strong>
changed from <em>new</em> to <em>assigned</em>
</li>
</ul>
<p>
Stephen Blackheath ran the ddc building the Rover project under the profiler which show that huge amounts of memory were being chewed up the functions scrapeGraphInsert and invalidateParents when DDC figures out which files need to be rebuilt.
</p>
<p>
After this stage the compiler is actually quite well behaved.
</p>
TicketerikdMon, 01 Feb 2010 04:59:36 GMTcomment added; attachment sethttp://trac.haskell.org/ddc/ticket/99
http://trac.haskell.org/ddc/ticket/99
<ul>
<li><strong>attachment</strong>
set to <em>ddc_profile.png</em>
</li>
</ul>
<p>
Profiler output showing huge memory usage of scapeGraphInsert and invalidateParents.
</p>
TicketerikdMon, 01 Feb 2010 10:26:07 GMTcomment added; attachment sethttp://trac.haskell.org/ddc/ticket/99
http://trac.haskell.org/ddc/ticket/99
<ul>
<li><strong>attachment</strong>
set to <em>ddc-strict-state.png</em>
</li>
</ul>
<p>
After switching to Control.Monad.State.Strict the profile output looks like this.
</p>
TicketerikdMon, 01 Feb 2010 10:49:06 GMTcomment addedhttp://trac.haskell.org/ddc/ticket/99#comment:3
http://trac.haskell.org/ddc/ticket/99#comment:3
<p>
It seems the code to detect cycles in the import graph is grossly inefficient. Since I added that code I'll fix this bug.
</p>
TicketerikdSat, 06 Feb 2010 11:50:50 GMTcomment added; summary changedhttp://trac.haskell.org/ddc/ticket/99#comment:4
http://trac.haskell.org/ddc/ticket/99#comment:4
<ul>
<li><strong>summary</strong>
changed from <em>Work out why test/90-Programs/Rover is crashing</em> to <em>Finding which modules needs to be rebuilt is grossly inefficient.</em>
</li>
</ul>
TicketerikdSat, 06 Feb 2010 20:45:37 GMTcomment added; resolution set; status changedhttp://trac.haskell.org/ddc/ticket/99#comment:5
http://trac.haskell.org/ddc/ticket/99#comment:5
<ul>
<li><strong>status</strong>
changed from <em>assigned</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
</ul>
<p>
Fixed in the following commit:
</p>
<ul><li>Sat Feb 6 07:03:35 EST 2010 Erik de Castro Lopo &lt;erikd@…&gt;
</li></ul><blockquote>
<p>
Fix <a class="closed ticket" href="http://trac.haskell.org/ddc/ticket/99" title="defect: Finding which modules needs to be rebuilt is grossly inefficient. (closed: fixed)">#99</a> : Finding which modules need to be rebuilt is inefficient.
</p>
</blockquote>
<p>
</p>
<blockquote>
<p>
Profiling of the build of the Rover program in the test suite show that as much
as 90% of the run time and 90% of the memory usage was used just to figure
out which modules needed to be rebuilt.
</p>
</blockquote>
<p>
</p>
<blockquote>
<p>
The problem turned out to be an O(n<sup>2</sup>) (and possibly worse) algorithm which
tried to build the <tt>ScrapeGraph</tt> and figure out which modules needed to be
rebuild in a single pass. Separating this into two passed, one to build the
<tt>ScrapeGraph</tt> and one to propagate the <tt>NeedsRebuild</tt> flag fixed the problem.
Building the <tt>ScrapeGraph</tt> now takes less than 1% of the compile run time.
</p>
</blockquote>
Ticket