Jekyll2018-10-24T11:57:32+00:00https://devopsvoyage.com/DevOps VoyageAn excerpt from day to day debugging.Leszek Zalewskihttps://devopsvoyage.comHow to: Execute RSpec in parallel locally2018-10-22T00:00:00+00:002018-10-24T12:00:00+00:00https://devopsvoyage.com/2018/10/22/execute-rspec-locally-in-parallel<p>Last entry in series about stable and faster test suite. Previous posts were focusing on parallel execution when running the test suite on CI nodes.
This time we will see how we can run it locally as well.
You can find previous articles here</p>
<ul>
<li><a href="/2018/09/12/road-to-fast-and-stable-test-suite.html" title="How to: Road to fast and stable test suite"><em>How to: Road to fast and stable test suite</em></a></li>
<li><a href="/2018/09/26/get-most-of-the-database-cleaner.html" title="How to: Get most of the database cleaner"><em>How to: Get most of the database cleaner</em></a></li>
<li><em>How to: Execute RSpec in parallel locally</em> (You are here)</li>
</ul>
<h1>ParallelTests gem</h1>
<p>So far we used <a href="https://docs.knapsackpro.com/ruby/knapsack" title="Knapsack Gem documentation page">Knapsack</a> gem for dividing our tests in order to evenly run them on CI,
but we can&rsquo;t really use it for running them all in parallel locally. This is where <a href="https://github.com/grosser/parallel_tests" title="ParallelTests source code and documentation"><code>ParallelTests</code></a> gem
will come in handy.</p>
<blockquote>
<p>ParallelTests splits tests into even groups (by the number of lines or runtime) and runs each group in a single process with its own database.</p>
</blockquote>
<p>So in short, each process spawned by ParallelTests should have its resources isolated from one another, so they don&rsquo;t interfere with each other.</p>
<h2>Basic setup</h2>
<p>Add parallel tests to development and test groups in Gemfile.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">gem</span> <span class="s1">'parallel_tests'</span><span class="p">,</span> <span class="ss">group: </span><span class="sx">%i[development test]</span>
</code></pre></div>
<p>Next, let us decide on the number of processors You want to use. By default <code>parallel_tests</code> will set it to
number of CPUs available (i.e. 4 cores with hyperthreading, will count as 8 cores).
If You would like to override it, append <code>[&lt;no of processors to run&gt;]</code> to tasks below or better,
export environment variable to have it the same for all the tasks, i.e.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">PARALLEL_TEST_PROCESSORS</span><span class="o">=</span>8
</code></pre></div>
<p>Just remember to put it in something like <code>.bashrc</code> or <code>.env</code> file, in order to have it always loaded between sessions.</p>
<h2>Resources configuration</h2>
<h3>1. Database</h3>
<p>I assume You use a database, otherwise, you wouldn&rsquo;t have issues with slow running tests in the first place.
For this to work, we need to update our <code>config/database.yml</code> with an extended database name.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">test</span><span class="pi">:</span>
<span class="na">database</span><span class="pi">:</span> <span class="s">yourproject_test&lt;%= ENV['TEST_ENV_NUMBER'] %&gt;</span>
</code></pre></div>
<p>This will append the processor number to your database name, whenever you will run tests with more than 1 process, i.e.</p>
<ol>
<li>First process <code>yourproject_test</code></li>
<li>Second process <code>yourproject_test1</code></li>
<li>etc</li>
</ol>
<p>Now let&rsquo;s create all databases.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">rake parallel:create
</code></pre></div>
<p>And load the <code>schema.rb</code> or <code>structure.sql</code> to all DBs created above.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">rake parallel:prepare
</code></pre></div>
<h3>2. Capybara</h3>
<p>Capybara servers should run on separate ports, i.e.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">Capybara</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">server_port</span> <span class="o">=</span> <span class="mi">9887</span> <span class="o">+</span> <span class="no">ENV</span><span class="p">[</span><span class="s1">'TEST_ENV_NUMBER'</span><span class="p">].</span><span class="nf">to_i</span>
<span class="k">end</span>
</code></pre></div>
<h3>3. Other resources</h3>
<p>You should do the same for any other kind of resource You use, those should be completely separated from each other, i.e.:</p>
<ul>
<li>files, i.e. for rails/sprockets cache should lay in different directories</li>
<li>redis, use different DBs per process</li>
<li>sphinx, run a separate instance per each processor</li>
<li>etc</li>
</ul>
<p>Checkout extensive <a href="https://github.com/grosser/parallel_tests/wiki" title="ParallelTests wiki page with documentation about various resources setup">wiki</a> for details.</p>
<h2>Lets run this!</h2>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">rake parallel:spec
<span class="c"># =&gt; 8 processes for 500 specs, ~ 62 specs per process</span>
</code></pre></div>
<figure>
<img alt="Specs using full power of all CPUs via htop tool"src="/assets/images/2018-10-22/cpu-load.gif"/>
<figcaption>Full throttle!</figcaption>
</figure>
<p>Awesome! Now we can run our test suite locally as fast as on CI.</p>
<h2>Speedup process boot</h2>
<p>Now when spawning all those processes, each of them will take some time, based on the size of your application.
We can speed this up with <code>spring</code>, which comes by default with rails installations these days.
To make it work we will have to do a small patch for it, otherwise it won&rsquo;t work with <code>parallel_tests</code>.</p>
<p>This is due to the fact that with <code>spring</code>, when it boot up the server process, the configuration will be already set.
This means DB name will always equal to <code>yourproject_test</code> in each spawned process based on the server one.</p>
<p>In order to mitigate this, we will pick up the correct DB configuration
after forking the server process. You can find it in <a href="https://github.com/grosser/parallel_tests/wiki" title="ParallelTests wiki page with documentation about various resources setup"><code>parallel_tests</code> Wiki</a>.</p>
<p>Create new file under <code>config/spring.rb</code> - it should get picked up by spring automatically.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s1">'spring/application'</span>
<span class="k">class</span> <span class="nc">Spring</span><span class="o">::</span><span class="no">Application</span>
<span class="k">alias</span> <span class="n">connect_database_orig</span> <span class="n">connect_database</span>
<span class="c1"># Disconnect &amp; reconfigure to pickup DB name with</span>
<span class="c1"># TEST_ENV_NUMBER suffix</span>
<span class="k">def</span> <span class="nf">connect_database</span>
<span class="n">disconnect_database</span>
<span class="n">reconfigure_database</span>
<span class="n">connect_database_orig</span>
<span class="k">end</span>
<span class="c1"># Here we simply replace existing AR from main spring process</span>
<span class="k">def</span> <span class="nf">reconfigure_database</span>
<span class="k">if</span> <span class="n">active_record_configured?</span>
<span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span><span class="p">.</span><span class="nf">configurations</span> <span class="o">=</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">config</span><span class="p">.</span><span class="nf">database_configuration</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>Then you can either prepend <code>DISABLE_SPRING=0</code> to commands, export it, or put in <code>.bashrc</code>/<code>.env</code> file.</p>
<p>With this, <code>bin/rake parallel:spec</code> will boot up way faster by making use of <code>spring</code> preloader.
You should notice this by extra lines in output like</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">Running via Spring preloader <span class="k">in </span>process 20005
Running via Spring preloader <span class="k">in </span>process 20012
Running via Spring preloader <span class="k">in </span>process 20015
... etc
</code></pre></div>
<h2>Tests distribution</h2>
<p>Another thing to consider is, how to evenly distribute tests across processes.
<code>parallel_tests</code> has similar functionality as <code>knapsack</code>, it can log tests runtime in a JSON file and then use it to spread them evenly across all processes.
To do so add <code>.rspec_parallel</code> file in project root directory, so on next run it will create the report, and use it in consecutive executions.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nt">--format</span> progress
<span class="nt">--format</span> ParallelTests::RSpec::RuntimeLogger <span class="nt">--out</span> tmp/parallel_runtime_rspec.log
</code></pre></div>
<p><strong>NOTE</strong>: Remember to put any significat config from <code>.rspec</code> file to <code>.rspec_parallel</code>, i.e. <code>--require spec_helper</code> -
as parallel tests will use the later only, what can lead to issues with tests.</p>
<p>Now, this is good for having local runtime as low as possible, but what if we would like to use the knapsack report, which we already have?
Sadly <code>parallel_tests</code> doesn&rsquo;t have any integration for it, but we can play around and add it by ourselves - because we can :)</p>
<p>Let&rsquo;s create a wrapper task we will use to run it</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># lib/tasks/knapsack.rake</span>
<span class="n">namespace</span> <span class="ss">:knapsack</span> <span class="k">do</span>
<span class="n">task</span> <span class="ss">local: :environment</span> <span class="k">do</span> <span class="o">|</span><span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="o">|</span>
<span class="no">ENV</span><span class="p">[</span><span class="s1">'CI_NODE_TOTAL'</span><span class="p">]</span> <span class="o">=</span> <span class="no">ENV</span><span class="p">[</span><span class="s1">'PARALLEL_TEST_PROCESSORS'</span><span class="p">]</span>
<span class="no">ENV</span><span class="p">[</span><span class="s1">'RAILS_ENV'</span><span class="p">]</span> <span class="o">=</span> <span class="s1">'test'</span>
<span class="nb">require</span> <span class="s1">'knapsack'</span>
<span class="nb">require_relative</span> <span class="s1">'../../config/boot'</span>
<span class="nb">require_relative</span> <span class="s1">'parallel_tests_patch'</span>
<span class="no">ParallelTests</span><span class="o">::</span><span class="no">CLI</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">run</span><span class="p">([</span><span class="s1">'--type'</span><span class="p">,</span> <span class="s1">'rspec'</span><span class="p">,</span> <span class="s1">'spec'</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>Now a small monkey patch to use knapsack allocator in parallel tests</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># lib/tasks/parallel_tests_patch.rb</span>
<span class="nb">require</span> <span class="s1">'parallel_tests/rspec/runner'</span>
<span class="k">class</span> <span class="nc">ParallelTests</span><span class="o">::</span><span class="no">RSpec</span><span class="o">::</span><span class="no">Runner</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">tests_in_groups</span><span class="p">(</span><span class="n">tests</span><span class="p">,</span> <span class="n">num_groups</span><span class="p">,</span> <span class="n">options</span> <span class="o">=</span> <span class="p">{})</span>
<span class="nb">puts</span> <span class="s1">'ParallelTests with Knapsack runtime report :woohoo:'</span>
<span class="p">(</span><span class="mi">0</span><span class="o">...</span><span class="n">num_groups</span><span class="p">).</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">index</span><span class="o">|</span>
<span class="no">ENV</span><span class="p">[</span><span class="s1">'CI_NODE_INDEX'</span><span class="p">]</span> <span class="o">=</span> <span class="n">index</span><span class="p">.</span><span class="nf">to_s</span>
<span class="no">Knapsack</span><span class="o">::</span><span class="no">AllocatorBuilder</span>
<span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="no">Knapsack</span><span class="o">::</span><span class="no">Adapters</span><span class="o">::</span><span class="no">RSpecAdapter</span><span class="p">)</span>
<span class="p">.</span><span class="nf">allocator</span>
<span class="p">.</span><span class="nf">node_tests</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>Let&rsquo;s test it!</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash">rake knapsack:local
<span class="c"># ParallelTests with Knapsack runtime report :woohoo:</span>
<span class="c"># 8 processes for 500 specs, ~ 62 specs per process</span>
</code></pre></div>
<p><strong>NOTE</strong> For local execution I would still use <code>parallel_tests</code> allocator instead, as it will be generated based on our machine performance, whereas knapsack is supposed to be based on the CI node.</p>
<h1>Runtime comparison</h1>
<table><thead>
<tr>
<th>Processes</th>
<th>Spring?</th>
<th>Runtime log?</th>
<th>Runtime</th>
</tr>
</thead><tbody>
<tr>
<td>8</td>
<td>yes</td>
<td>yes</td>
<td><code>8m 22.928s</code></td>
</tr>
<tr>
<td>7</td>
<td>yes</td>
<td>yes</td>
<td><code>8m 55.850s</code></td>
</tr>
<tr>
<td>8</td>
<td>no</td>
<td>yes</td>
<td><code>8m 57.309s</code></td>
</tr>
<tr>
<td>6</td>
<td>yes</td>
<td>yes</td>
<td><code>10m 03.678s</code></td>
</tr>
<tr>
<td><em>9</em></td>
<td>yes</td>
<td>yes</td>
<td><code>10m 09.448s</code></td>
</tr>
<tr>
<td>8</td>
<td>yes</td>
<td>no</td>
<td><code>12m 59.819s</code></td>
</tr>
<tr>
<td>1</td>
<td>no</td>
<td>no</td>
<td><code>42m 00.501s</code></td>
</tr>
</tbody></table>
<p>As you can see above, our suite runtime without any parallelization takes quite a while, around ~42 minutes.</p>
<p>When we parallelize it with 8 processes result differs based on extra switches.</p>
<p>Without the runtime log to evenly distribute tests, take the longest (even with spring support).</p>
<p>We can also see that having more than 8 processes is also degrading runtime.</p>
<p>In our case the best results are achieved when:</p>
<ul>
<li>running against 8 processes on 8 available cores</li>
<li>run together with runtime logs for tests distribution (<em>-4&#39;30&quot;</em>)</li>
<li>processes are preloaded by spring (<em>-30&quot;</em>)</li>
</ul>
<p>This can be due to the fact that we have a lot of IO in tests, in tests and appications that do heavy computing, less
processes can actually give better results. As usuall measure, compare and take the most performant option ;-)</p>
<h1>Summary and what&rsquo;s next</h1>
<p>From now on it should be easy to run Your test suite in parallel, both locally and on CI with help of <code>knapsack</code> and <code>parallel_tests</code> gems.</p>
<p>The test suite itself should be also faster and more stable thanks to better usage of <code>DatabaseCleaner</code>.</p>
<p>Thankfully Rails 6 should bring us built-in support for</p>
<ul>
<li>running tests in parallel</li>
<li>better transactions handling</li>
<li>and multi-database connection support</li>
</ul>
<p>So we won&rsquo;t have to hack our way through in new apps, for old ones running on Rails <code>&lt;= 5</code> we are already covered.</p>
<p><em>Stay tuned and happy hacking!</em></p>Leszek Zalewskihttps://devopsvoyage.comPrevious posts were focusing on parallel execution when running the test suite on CI nodes. This time we will see how we can run it locally as well.How to: Get most of the database cleaner2018-09-26T00:00:00+00:002018-09-27T09:52:28+00:00https://devopsvoyage.com/2018/09/26/get-most-of-the-database-cleaner<p>In the <a href="/2018/09/12/road-to-fast-and-stable-test-suite.html" title="How to: Road to fast and stable test suite">previous post</a>, we saw how we can divide and speed up test suite by using <a href="https://docs.knapsackpro.com/ruby/knapsack" title="Knapsack Gem documentation page">Knapsack gem</a> from ~1 hour to 11 minutes.
We also cached our dependencies on CI and started looking out for randomly failing tests.
This time we will take a look at how to setup DatabaseCleaner in order to make use of different
strategies for wiping out data between test runs in order to squeeze in even more from the suite.</p>
<h1>DatabaseCleaner gem</h1>
<p><a href="https://github.com/DatabaseCleaner/database_cleaner" title="DatabaseCleaner repository page on Github">DatabaseCleaner</a> is a set of strategies for cleaning your database in Ruby.
It&rsquo;s a simple yet powerful tool that You most probably already use. Even though it&rsquo;s simple,
its configuration can get tricky at times. Messing up configuration can cause extra failures
related to database pollution. So let us see how we can set it up in order to have a stable test suite.</p>
<h2>Basic setup</h2>
<p>Add database cleaner to test group in Gemfile (if it&rsquo;s not yet there).</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">group</span> <span class="ss">:test</span> <span class="k">do</span>
<span class="n">gem</span> <span class="s1">'database_cleaner'</span>
<span class="k">end</span>
</code></pre></div>
<p>Make sure you load <em>rspec/support</em> helpers, it&rsquo;s loaded by default with RSpec setup in <code>spec/rails_helper</code>.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># depending on age of the project look into rails or spec helper file</span>
<span class="no">Dir</span><span class="p">[</span><span class="no">Rails</span><span class="p">.</span><span class="nf">root</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="s1">'spec'</span><span class="p">,</span> <span class="s1">'support'</span><span class="p">,</span> <span class="s1">'**'</span><span class="p">,</span> <span class="s1">'*.rb'</span><span class="p">)].</span><span class="nf">each</span> <span class="p">{</span> <span class="o">|</span><span class="n">f</span><span class="o">|</span> <span class="nb">require</span> <span class="n">f</span> <span class="p">}</span>
</code></pre></div>
<p>Create <code>spec/support/database_cleaner.rb</code> with</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">RSpec</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:suite</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean_with</span><span class="p">(</span><span class="ss">:deletion</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>This is a good starting point, it will make sure the database is clean before the whole suite starts in case
there are any leftovers from previous run.</p>
<h2>Transactions - fastest all-rounder</h2>
<p><strong>Transactions</strong> should be used by default, when you need to use database calls it&rsquo;s the fastest strategy available.
DB Cleaner will open new transaction on each test case and roll it back when it&rsquo;s finished. This way it doesn&rsquo;t
need to truncate all the tables, even though You used just one.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">RSpec</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="c1"># ...</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">strategy</span> <span class="o">=</span> <span class="ss">:transaction</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">after</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h2>Deletion - rescue for after commit callbacks</h2>
<p>Sometimes transactions are not enough, especially when you use after commit callbacks in ActiveRecord.
As the name implies those are executed after committing the transaction. This can&rsquo;t work with the previous
strategy as transaction would still be open on the DB Cleaner level.</p>
<p>For this kind of unit tests where you touch DB and need access to data after committing the transaction,
you can use <strong>deletion</strong> strategy. Which won&rsquo;t be significantly slower than the <em>transaction</em> for small data sets.</p>
<p>To fix this up without switching all the tests to <em>deletion</em>, we can use RSpec examples metadata,
in order to pick correct strategy on demand.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"> <span class="c1"># ...</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">strategy</span> <span class="o">=</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:strategy</span><span class="p">]</span> <span class="o">||</span> <span class="ss">:transaction</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="c1"># ...</span>
</code></pre></div>
<p>With this whenever we need to use the deletion strategy, we will just add <code>strategy: :deletion</code>, like:</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># for whole block</span>
<span class="n">context</span> <span class="s2">"executes after commit callbacks"</span><span class="p">,</span> <span class="ss">strategy: :deletion</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="c1"># or a single example</span>
<span class="n">it</span> <span class="s2">"creates thumbnails on save"</span><span class="p">,</span> <span class="ss">strategy: :deletion</span> <span class="k">do</span>
<span class="c1"># ...</span>
<span class="k">end</span>
</code></pre></div>
<h2>Truncation - safe acceptance examples</h2>
<p><strong>Truncation</strong> is the <em>slowest</em> but also the <em>safest</em> and most stable strategy.
There are no compromises when it comes to clean state here.
It&rsquo;s the best strategy for acceptance tests when using capybara with javascript driver.</p>
<p>Again we can make use of metadata, and use it by default for all js related tests.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"> <span class="c1"># ...</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">strategy</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:js</span><span class="p">]</span>
<span class="ss">:truncation</span>
<span class="k">else</span>
<span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:strategy</span><span class="p">]</span> <span class="o">||</span> <span class="ss">:transaction</span>
<span class="k">end</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="c1"># ... </span>
</code></pre></div>
<h2>Truncation or Deletion?</h2>
<p>Rule of thumb should be:</p>
<p>For big setups before the tests, when you create a lot of objects with associations that use foreign keys, etc -
use <strong>truncation</strong>, as it&rsquo;s a fixed time regardless of the amount of data. This also means the more tables you have
the more time it will take, as it takes the same amount of time for empty tables as well. You can think about it as
drop plus create table, indexes, etc.</p>
<p>On the other hand, when you have a simple setup, with unit alike tests - use <strong>deletion</strong> as it&rsquo;s faster for small
datasets, it won&rsquo;t recreate tables or indexes.</p>
<p>If you are interested in details, then <a href="https://stackoverflow.com/questions/11419536/postgresql-truncation-speed/11423886#11423886" title="Answer by Craig Ringer for &#39;Postgresql Truncation speed&#39; question on StackOverflow">here&rsquo;s a great answer for PostgreSQL on StackOverflow</a> explaining it thoroughly.</p>
<h2>Bonus no.1: Cleaning on demand</h2>
<p>Life is hard, and usually, you will find yourself with a really big setup of objects. This can happen especially
when you try to test some kind of filtering or querying classes, which do read-only queries against data,
without any side effects. In such cases, it would be great if we could disable DB Cleaner per example,
and run it only once in <code>before/after</code> all blocks. Again this is easy to achieve with examples metadata, lets us see.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"> <span class="c1"># before :suite ...</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:all</span><span class="p">,</span> <span class="ss">:cleaner_for_context</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">strategy</span> <span class="o">=</span> <span class="ss">:truncation</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">example</span><span class="o">|</span>
<span class="k">next</span> <span class="k">if</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:cleaner_for_context</span><span class="p">]</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">strategy</span> <span class="o">=</span> <span class="c1"># ...</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">after</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">example</span><span class="o">|</span>
<span class="k">next</span> <span class="k">if</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:cleaner_for_context</span><span class="p">]</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">after</span><span class="p">(</span><span class="ss">:all</span><span class="p">,</span> <span class="ss">:cleaner_for_context</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean</span>
<span class="k">end</span>
<span class="c1"># ...</span>
</code></pre></div>
<p>Lets go into detail what do we do here:</p>
<ol>
<li><code>before :all, :cleaner_for_context</code> block, will run once for whole context/describe block. We used truncation, as it will usually be faster in such examples.</li>
<li>We skip <code>before</code> and <code>after :each</code> blocks whenever we set <code>:cleaner_for_context</code> metadata.</li>
<li>We clean database once for whole context/describe block with <code>after :all</code></li>
</ol>
<p>Then in our test, we can do</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">RSpec</span><span class="p">.</span><span class="nf">describe</span> <span class="no">CoreDashboardQuery</span><span class="p">,</span> <span class="ss">:cleaner_for_context</span> <span class="k">do</span>
<span class="n">before</span> <span class="ss">:all</span> <span class="k">do</span>
<span class="vi">@data</span> <span class="o">=</span> <span class="c1"># ...</span>
<span class="k">end</span>
<span class="n">let</span><span class="p">(</span><span class="ss">:data</span><span class="p">)</span> <span class="p">{</span> <span class="vi">@data</span> <span class="p">}</span>
<span class="c1"># it ...</span>
<span class="k">end</span>
</code></pre></div>
<p><strong>Note</strong>: Remember that whenever you use before/after all blocks, those are run <strong>once</strong> for the given context/describe block, so:</p>
<ul>
<li>don&rsquo;t do any data updates or additions within those blocks, unless you want to shoot yourself in the foot with a canon</li>
<li>whenever you use this technique, create the setup on top of the file, don&rsquo;t hide it somewhere nested on the bottom</li>
</ul>
<h2>Bonus no.2: Catch data pollution issues</h2>
<p>Getting back to our test suite example, which is still <em>dark and full of terrors</em>. We had random failures that were related to test
leftovers in the database. Those were caused in examples that were running with <code>:transaction</code> strategy, were <code>before :all</code> block
was lurking in the depths of nested context blocks.</p>
<p>The easiest way to find those is to crash loudly whenever the database is not clean in <code>after :all</code> block. To know what to count,
look out for errors like <em>&ldquo;Couldn&rsquo;t create Project because of unique key validation&rdquo;</em> or similar. To catch those drop this line in our setup.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="c1"># still in our spec/support/database_cleaner.rb file</span>
<span class="no">RSpec</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="k">class</span> <span class="nc">DirtyDatabaseError</span> <span class="o">&lt;</span> <span class="no">RuntimeError</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">meta</span><span class="p">)</span>
<span class="k">super</span> <span class="s2">"</span><span class="si">#{</span><span class="n">meta</span><span class="p">[</span><span class="ss">:full_description</span><span class="p">]</span><span class="si">}</span><span class="se">\n\t</span><span class="si">#{</span><span class="n">meta</span><span class="p">[</span><span class="ss">:location</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># ...</span>
<span class="n">config</span><span class="p">.</span><span class="nf">after</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">example</span><span class="o">|</span>
<span class="k">next</span> <span class="k">if</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:cleaner_for_context</span><span class="p">]</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean</span>
<span class="k">raise</span> <span class="no">DirtyDatabaseError</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">)</span> <span class="k">if</span> <span class="no">Project</span><span class="p">.</span><span class="nf">count</span> <span class="o">&gt;</span> <span class="mi">0</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>This will crash as soon as there are records still in DB, and will print example which caused it:</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh">Failures:
1<span class="o">)</span> ProjectsQuery#build_query sorts projects by title
Failure/Error: raise DirtyDatabaseError.new<span class="o">(</span>example.metadata<span class="o">)</span> <span class="k">if </span>Project.count <span class="o">&gt;</span> 0
DirtyDatabaseError:
ProjectsQuery#build_query sorts projects by title
./spec/queries/projects_query_spec.rb:278
</code></pre></div>
<p>Of course, you don&rsquo;t want to run it every time,
so just drop it in whenever You see some suspicious errors.</p>
<h1>Summary</h1>
<p>Our final DatabaseCleaner configuration for RSpec.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="k">class</span> <span class="nc">DirtyDatabaseError</span> <span class="o">&lt;</span> <span class="no">RuntimeError</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">meta</span><span class="p">)</span>
<span class="k">super</span> <span class="s2">"</span><span class="si">#{</span><span class="n">meta</span><span class="p">[</span><span class="ss">:full_description</span><span class="p">]</span><span class="si">}</span><span class="se">\n\t</span><span class="si">#{</span><span class="n">meta</span><span class="p">[</span><span class="ss">:location</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">RSpec</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:suite</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean_with</span><span class="p">(</span><span class="ss">:deletion</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:all</span><span class="p">,</span> <span class="ss">:cleaner_for_context</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">strategy</span> <span class="o">=</span> <span class="ss">:truncation</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">before</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">example</span><span class="o">|</span>
<span class="k">next</span> <span class="k">if</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:cleaner_for_context</span><span class="p">]</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">strategy</span> <span class="o">=</span>
<span class="k">if</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:js</span><span class="p">]</span>
<span class="ss">:truncation</span>
<span class="k">else</span>
<span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:strategy</span><span class="p">]</span> <span class="o">||</span> <span class="ss">:transaction</span>
<span class="k">end</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">start</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">after</span><span class="p">(</span><span class="ss">:each</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">example</span><span class="o">|</span>
<span class="k">next</span> <span class="k">if</span> <span class="n">example</span><span class="p">.</span><span class="nf">metadata</span><span class="p">[</span><span class="ss">:cleaner_for_context</span><span class="p">]</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean</span>
<span class="c1"># raise DirtyDatabaseError.new(example.metadata) if Record.count &gt; 0</span>
<span class="k">end</span>
<span class="n">config</span><span class="p">.</span><span class="nf">after</span><span class="p">(</span><span class="ss">:all</span><span class="p">,</span> <span class="ss">:cleaner_for_context</span><span class="p">)</span> <span class="k">do</span>
<span class="no">DatabaseCleaner</span><span class="p">.</span><span class="nf">clean</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>In our case, we were using <code>:truncation</code> for everything. In order to speed things up, we added above test configuration to use <code>:transaction</code>
by default for everything, and we left <code>:truncation</code> for JS tests only.</p>
<p>We also used <code>:deletion</code> instead of <code>:truncation</code> for unit alike tests,
that used after commit callbacks, i.e. attachment objects that are processing files when it&rsquo;s saved. In those cases, the deletion was taking a
fraction of the time of truncation.</p>
<p>On the way, we found out which tests were polluting database, and we were able to fix it easily. When we
finished the move, our suite runtime dropped by another 2 minutes and now stays stable at ~9 minutes per job.</p>
<p>Next time we will try making use of knapsack to run the test suite in parallel locally, so <em>stay tuned and happy hacking!</em></p>Leszek Zalewskihttps://devopsvoyage.comThe second part from fast and stable test suite series. We will see how various strategies for cleaning the database in tests can speed up test suite even more.How to: Road to fast and stable test suite2018-09-12T00:00:00+00:002018-09-27T09:52:28+00:00https://devopsvoyage.com/2018/09/12/road-to-fast-and-stable-test-suite<blockquote>
<p><em>It&rsquo;s an extension for my presentation from Ruby User Group Berlin meetup in September, 2018, <a href="https://slides.com/zalesz/testing-sanity-p1">You can find slides here</a>.</em></p>
</blockquote>
<p>A beginning of the story on how we speed up test suite in one of the projects at Akelius from one hour to less than 10 minutes.</p>
<p><strong>TL;DR</strong>: <em>Divide and Conquer</em></p>
<ul>
<li>start with making tests run in <em>parallel</em> in the easiest possible manner</li>
<li>start tracking tests that fail in each of the subsets of tests</li>
<li>reproduce and fix issues locally</li>
</ul>
<h2>Why?</h2>
<p>I don&rsquo;t think it needs to be said out loud but here it goes - <em>slow and flaky test suites are bad for you!</em>
If Your suite runs for <em>over</em> 10 minutes you are hurting yourself and your team.
But let us start from the beginning. When I started in Akelius, application test suite was running for an hour.
Yes, you read it correctly - an <strong>HOUR</strong>. It also had order dependant test failures, so 1 in 4 builds could randomly fail.</p>
<figure>
<img alt="Slow, failed build on Travis CI" src="/assets/images/2018-09-12/01-slow-failure.png"/>
<figcaption>Oh no, not again!</figcaption>
</figure>
<p>By using Github checks, that CI needs to pass in order to merge a Pull Request, this would basically block the deployment to next stages.
Due to a random failure, the build would be usually restarted in order to move forward - which doesn&rsquo;t mean it would be green on the next run.</p>
<figure>
<img alt="Slow, successfull build on Travis CI" src="/assets/images/2018-09-12/02-slow-success.png" />
<figcaption>Finally, some time later, after 1-3 restarts.</figcaption>
</figure>
<p>This resulted in:</p>
<ul>
<li>very long feedback loops</li>
<li>lots of distractions due to context switching</li>
<li>frustration</li>
<li>and a useless test suite, which was more of a drag than a help</li>
<li>cost us lots of time</li>
</ul>
<p>So how did we handle this? We had to speed it up first, in order to easier track down random test failures.
How to do it with least effort in the ruby world?</p>
<h2>Enter: knapsack gem</h2>
<figure>
<img alt="Ruby gem 'Knapsack' logo" src="/assets/images/2018-09-12/03-knapsack.png" />
<figcaption>Knapsack splits tests across CI nodes and makes sure that tests will run comparable time on each node.</figcaption>
</figure>
<p><a href="https://docs.knapsackpro.com/ruby/knapsack" title="Knapsack Gem documentation page">Knapsack</a> doesn&rsquo;t require any extra setup for additional services you are using, like database or chrome.
Instead, it relies completely on the isolation provided by CI nodes.
The only logic involved is <strong>how to evenly divide test files across nodes, so nodes run time is balanced</strong>,
i.e. all nodes finish in ~5 minutes, instead of one finishes in 2 minutes and another one in 8.</p>
<p>Installation is quite easy and documentation covers a variety of ruby test runners.</p>
<h2>Parallelize tests with the knapsack</h2>
<p>Add gem in Gemfile.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="n">group</span> <span class="ss">:test</span><span class="p">,</span> <span class="ss">:development</span> <span class="k">do</span>
<span class="n">gem</span> <span class="s2">"knapsack"</span>
<span class="k">end</span>
</code></pre></div>
<p>Add tasks in Rakefile.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="no">Knapsack</span><span class="p">.</span><span class="nf">load_tasks</span> <span class="k">if</span> <span class="n">defined?</span><span class="p">(</span><span class="no">Knapsack</span><span class="p">)</span>
</code></pre></div>
<p>Bind knapsack in top of spec helper file.</p>
<div class="highlight"><pre><code class="language-ruby" data-lang="ruby"><span class="nb">require</span> <span class="s2">"knapsack"</span>
<span class="no">Knapsack</span><span class="o">::</span><span class="no">Adapters</span><span class="o">::</span><span class="no">RSpecAdapter</span><span class="p">.</span><span class="nf">bind</span>
</code></pre></div>
<p>Generate report, so knapsack will know runtime of the specific test files and will be able to evenly split them across nodes.
It&rsquo;s <strong>best to run it on actual CI node</strong>, so results will be closest to real ones.</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="c"># locally</span>
<span class="nv">$&gt;</span> <span class="nv">KNAPSACK_GENERATE_REPORT</span><span class="o">=</span><span class="nb">true </span>bundle <span class="nb">exec </span>rspec spec
</code></pre></div><div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># or on CI, i.e. edit .travis.yml file</span>
<span class="na">script</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">KNAPSACK_GENERATE_REPORT=true</span><span class="nv"> </span><span class="s">bundle</span><span class="nv"> </span><span class="s">exec</span><span class="nv"> </span><span class="s">rspec</span><span class="nv"> </span><span class="s">spec"</span>
</code></pre></div>
<p>When the build completes, it will print out the new JSON report which will look something like:</p>
<div class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="s2">"spec/models/supply_spec.rb"</span><span class="p">:</span><span class="w"> </span><span class="mf">0.7876174449920654</span><span class="p">,</span><span class="w">
</span><span class="s2">"spec/models/tax_spec.rb"</span><span class="p">:</span><span class="w"> </span><span class="mf">0.22003436088562012</span><span class="p">,</span><span class="w">
</span><span class="s2">"spec/models/text_document_spec.rb"</span><span class="p">:</span><span class="w"> </span><span class="mf">3.3623762130737305</span><span class="p">,</span><span class="w">
</span><span class="s2">"spec/models/user_spec.rb"</span><span class="p">:</span><span class="w"> </span><span class="mf">318.7685122489929</span><span class="p">,</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div>
<p>Commit report to the repository in <code>knapsack_rspec_report.json</code> file.</p>
<p>The last step is to actually configure CI setup.
In our case it was TravisCI, but you can find docs for CircleCI and others as well, on <a href="https://docs.knapsackpro.com/ruby/knapsack" title="Knapsack Gem documentation page">knapsack documentation page</a>.
Here we say that we run our tests across 8 workers.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">env</span><span class="pi">:</span>
<span class="na">global</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">RAILS_ENV=test</span>
<span class="pi">-</span> <span class="s">CI_NODE_TOTAL=8</span> <span class="c1"># total number of workers</span>
<span class="na">matrix</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">CI_NODE_INDEX=0</span>
<span class="pi">-</span> <span class="s">CI_NODE_INDEX=1</span>
<span class="c1"># ... 5 more vars</span>
<span class="pi">-</span> <span class="s">CI_NODE_INDEX=7</span>
<span class="na">script</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s2">"</span><span class="s">bundle</span><span class="nv"> </span><span class="s">exec</span><span class="nv"> </span><span class="s">rake</span><span class="nv"> </span><span class="s">knapsack:rspec"</span>
</code></pre></div>
<p>Now lets watch them fly! 🚀</p>
<figure>
<img alt="Fast and successful test run on CI in under 12 minutes across 8 CI workers" src="/assets/images/2018-09-12/04-first-time-flying-tests.png"/>
<figcaption>First runtime improvement, from ~60 to ~12 minutes.</figcaption>
</figure>
<h3>Bonus #1: Outsource linters</h3>
<p>In our case we were using linters during CI run. By moving to CodeClimate, we shaved another ~30 seconds per build.</p>
<p>Afterwards we could let go 4 gems from Gemfile, which reduced total number of dependencies by 11.</p>
<h3>Bonus #2: Use headless chrome and cache dependencies</h3>
<p>Always try to prefer dependencies, that come preloaded on CI machines and make use of them.
This will usually result in extra bonus seconds, that can be spent running suite, instead of installing dependencies by hand.</p>
<p>One of such dependencies is chrome, which we use for acceptance tests.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># .travis.yml</span>
<span class="c1"># remove</span>
<span class="na">before_install</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">sudo apt-get install chromium-browser</span>
<span class="c1"># add</span>
<span class="na">addons</span><span class="pi">:</span>
<span class="na">chrome</span><span class="pi">:</span> <span class="s">stable</span>
</code></pre></div>
<p>And if you are using headless chrome, instead of <code>xvfb</code>, you need to download the chromedriver for it to work.
We make use of builtin caching mechanism, to download it, only if it&rsquo;s missing in the cached directory.
With this we saved extra ~20s per build. And we can easily install newer version if needed.</p>
<div class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># .travis.yml</span>
<span class="na">cache</span><span class="pi">:</span>
<span class="na">bundler</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">directories</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">~/bin</span>
<span class="na">before_install</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">./bin/ci_install_chromedriver</span>
</code></pre></div>
<p>In <code>bin/ci_install_chromedriver</code> we specify the script for checking and installing chromedriver.
We test against specific version, which will help invalidate the cache when we update.
You can use something simillar for other dependencies as well.</p>
<div class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/bash</span>
<span class="nb">set</span> <span class="nt">-Eeuo</span> pipefail
<span class="nv">VERSION</span><span class="o">=</span><span class="s2">"2.41"</span>
<span class="nv">NAME</span><span class="o">=</span><span class="s2">"chromedriver_</span><span class="k">${</span><span class="nv">VERSION</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">STORAGE</span><span class="o">=</span>https://chromedriver.storage.googleapis.com
install_chromedriver<span class="o">()</span> <span class="o">{</span>
<span class="nb">rm</span> <span class="nt">-f</span> ~/bin/chromedriver<span class="k">*</span>
wget <span class="k">${</span><span class="nv">STORAGE</span><span class="k">}</span>/<span class="k">${</span><span class="nv">VERSION</span><span class="k">}</span>/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
<span class="nb">rm </span>chromedriver_linux64.zip
<span class="nb">mv</span> <span class="nt">-f</span> chromedriver ~/bin/<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span>
<span class="nb">chmod</span> +x ~/bin/<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span>
<span class="nb">ln</span> <span class="nt">-s</span> ~/bin/<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span> ~/bin/chromedriver
<span class="o">}</span>
<span class="o">[</span> <span class="nt">-e</span> ~/bin/<span class="k">${</span><span class="nv">NAME</span><span class="k">}</span> <span class="o">]</span> <span class="o">||</span> install_chromedriver
</code></pre></div>
<h2>Track and fix random failures</h2>
<p>Since we now have fast running test suite, we don&rsquo;t only fail fast, we also reproduce the problem fast as well - as we now know the subset of tests that&rsquo;s causing the failure.
In order to reproduce those locally, we first need to find the job that failed, its <code>CI_NODE_INDEX</code>, and rspec order <code>seed</code>.
First one decides which files to run, second dictates in which order those are run.</p>
<figure>
<img alt="One of the 8 jobs failed on Travis CI" src="/assets/images/2018-09-12/05-flying-tests-fail.png" />
<figcaption>We now can run a subset of tests, which takes only ~8 minutes instead of 60.</figcaption>
</figure>
<p>After finding the job, you should be able to locate the <code>seed</code>, at the bottom of the failed test output.</p>
<figure>
<img alt="Bottom of failed test output listing failed tests case and showing rspec seed which was used for ordering tests"
src="/assets/images/2018-09-12/06-copy-rspec-seed.png" />
<figcaption>**4049** is what we are looking for.</figcaption>
</figure>
<p>Now, we can easily reproduce it locally, by running following command.</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$&gt;</span> <span class="nv">CI_NODE_TOTAL</span><span class="o">=</span>8 <span class="se">\</span>
<span class="nv">CI_NODE_INDEX</span><span class="o">=</span>5 <span class="se">\</span>
bundle <span class="nb">exec </span>rake <span class="s2">"knapsack:rspec[--seed 4049]"</span>
</code></pre></div>
<p>You can also try using <a href="https://relishapp.com/rspec/rspec-core/v/3-8/docs/command-line/bisect#use-%60--bisect%60-flag-to-create-a-minimal-repro-case-for-the-ordering-dependency" title="RSpec bisect flag documentation">RSpec <code>--bisect</code> flag</a>, to create a minimal repro case for the ordering dependency.
It will usually also find other issues on the way and take some time depending on the test suite.
Usually it&rsquo;s worth the wait, and if it&rsquo;s taking too long, you can always get results so far by hitting <code>CTRL+C</code>.</p>
<div class="highlight"><pre><code class="language-sh" data-lang="sh"><span class="nv">$&gt;</span> <span class="nv">CI_NODE_TOTAL</span><span class="o">=</span>8 <span class="se">\</span>
<span class="nv">CI_NODE_INDEX</span><span class="o">=</span>5 <span class="se">\</span>
bundle <span class="nb">exec </span>rake <span class="s2">"knapsack:rspec[--seed 4049 --bisect]"</span>
<span class="c"># =&gt;</span>
Report specs:
spec/models/user_spec.rb
<span class="c"># and a load of other tests scoped by knapsack</span>
Leftover specs:
Running via Spring preloader <span class="k">in </span>process 17947
Bisect started using options: <span class="s2">"--seed 4049 --default-path spec -- spec/models/user_spec.rb and a load of other tests scoped by knapsack"</span>
Running suite to find failures... <span class="o">(</span>4 minutes 23.9 seconds<span class="o">)</span>
Starting bisect with 6 failing examples and 921 non-failing examples.
Checking that failure<span class="o">(</span>s<span class="o">)</span> are order-dependent... failure<span class="o">(</span>s<span class="o">)</span> <span class="k">do </span>not require any non-failures to run first
Bisect <span class="nb">complete</span><span class="o">!</span> Reduced necessary non-failing examples from 921 to 0 <span class="k">in </span>12.51 seconds.
The minimal reproduction <span class="nb">command </span>is:
rspec ./spec/features/attachment_spec.rb[1:1:1,1:2,1:3] ./spec/features/project_spec.rb[1:1] ... <span class="nt">--seed</span> 4049 <span class="nt">--default-path</span> spec <span class="nt">--</span>
</code></pre></div>
<p>We had one example failing in the job, and we have found <em>6 in total</em> by running it with <code>--bisect</code> flag.</p>
<h2>Fix the issue and repeat</h2>
<p>This is subject for whole new post, or most probably, even a book. It will mostly depend on the test suite, but in general look out for:</p>
<ul>
<li>if using Rails, autoloading issues, especially when you have deeply nested structures like ActiveRecord models and Single Table Inheritance</li>
<li>using sleep statements, commonly found in acceptance tests</li>
<li>database pollution, i.e. some deeply nested <code>before :all</code> blocks, which run once for the whole file and not in the scope of nested describe/context blocks</li>
<li>make sure that the order of the elements doesn&rsquo;t play a role in tests that don&rsquo;t demand it, i.e. results in arrays</li>
</ul>
<p>If You can&rsquo;t fix them, as they happen, don&rsquo;t blindly restart the build. Instead note down the <code>CI_NODE_INDEX</code> and <code>seed</code>.
You can use some simple <em>TODO</em> list, but from my experience it&rsquo;s better to plan it properly.
For example, create maintenance ticket in JIRA (or in project management tool you are using), keep track of it, write down other failed examples.
So when the time comes, You can fix them in one go or assign as subtasks within the team.</p>
<h2>Profit &amp; Sanity</h2>
<p>After the first improvements, tests were running around 49 minutes faster per build, which resulted in:</p>
<ul>
<li>faster iterations on the tasks at hand</li>
<li>running the whole suite more often without the fear of failure</li>
<li>fewer distractions and context switching due to waiting for the results</li>
<li>team happiness and time to delivery :)</li>
</ul>
<p>There was one small drawback though - by using 8 workers, we block most of our available CI capacity.
You can probably already imagine, pushing one commit after another and blocking CI for everyone.
This is something to look out for, also maybe there are ways to cancel previous builds on the same branch.
We will investigate it in future when it becomes a bottleneck.</p>
<h2>Next steps</h2>
<p>First improvements resulted in a build time of around 11 minutes, in next article we will go over how we shaved another 2 minutes, by using a mix of different strategies for cleaning up a database.
We will also explore Jenkins, as a CI setup with kubernetes support and using <em>docker images</em> for build environments.</p>
<p><em>Stay tuned and happy hacking!</em></p>Leszek Zalewskihttps://devopsvoyage.comA beginning of the story on how we speed up test suite in one of the projects at Akelius from one hour to less than 10 minutes.