Jekyll2018-12-03T19:01:37+00:00https://josh.works/feed.xmlJosh Thompsongithub pages backend for www.josh.worksJosh Thompsonthompsonjoshd@gmail.comThe Complete Guide to Rails Performance: basic setup2018-11-29T13:00:00+00:002018-11-29T13:00:00+00:00https://josh.works/rails_speed_setup<p>You know the feeling.</p>
<p>You are excited to start a guide or a tutorial. You buy it, crack it open, and start working through the environment setup.</p>
<p>Then… something goes wrong. Next thing you know, you’ve spent <del>two</del> <del>three</del> too many hours debugging random crap, and you’re not even done with the introduction to the dang thing.</p>
<p>Oh, this has never happened to you? Must be nice.</p>
<p>I’m working through <a href="https://www.railsspeed.com/">The Complete Guide to Rails Performance</a>, and I’m thrilled to get learning underway.</p>
<p>I’ve hit a few hiccups, though. I’m not the most sophisticated user out there, so here’s a mess of problems I ran into, and the solutions I did.</p>
<h2 id="elastic-search">Elastic Search</h2>
<p>All of my struggles were with ElasticSearch. I’ve never used it before, so this isn’t surprising.</p>
<p><a href="https://github.com/rubygems/rubygems.org/blob/master/CONTRIBUTING.md#environment-os-x">The RubyGems docs</a> recommend the following:</p>
<blockquote>
<p>Install Elastic Search:</p>
<ul>
<li>Pull ElasticSearch 5.1.2 : <code class="highlighter-rouge">docker pull docker.elastic.co/elasticsearch/elasticsearch:5.1.2</code></li>
<li>Running Elasticsearch from the command line:
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:5.1.2
</code></pre></div> </div>
</li>
</ul>
</blockquote>
<!--more-->
<h2 id="docker-container-exit-code-137">Docker container, exit code 137</h2>
<p>The docker container was instantly exiting with a <a href="https://success.docker.com/article/what-causes-a-container-to-exit-with-code-137">137 exit code</a>. That means its out of memory.</p>
<p>I fixed this in the Docker app directly, but just bumping the memory allocated to the containers, and hitting the “save/restart” button.</p>
<p><img src="/images/2018-11-29 at 9.04 PM.jpg" alt="bump the memory" /></p>
<p>Now I could run:</p>
<p><code class="highlighter-rouge">bundle exec rake environment elasticsearch:import:all DIR=app/models FORCE=y</code></p>
<p>Which brought me to:</p>
<h2 id="elasticsearchunauthorized-401">Elasticsearch…Unauthorized: [401]</h2>
<p>I got</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Elasticsearch::Transport::Transport::Errors::Unauthorized: [401]
</code></pre></div></div>
<p>Sigh. Googled around, no quick solutions.</p>
<p>So, I hunted through the Slack group for the course, and saw someone suggest just <code class="highlighter-rouge">brew installing</code> ElasticSearch.</p>
<p>Done.</p>
<p><code class="highlighter-rouge">brew install elasticsearch</code></p>
<p>It installs <code class="highlighter-rouge">6.5.1</code> by default. This isn’t the recommended version, but… we’ll carry onward.</p>
<p>I can run <code class="highlighter-rouge">elasticsearch</code> in the terminal now, and get a lot of output.</p>
<p>Trying <code class="highlighter-rouge">bundle exec rake environment elasticsearch:import:all DIR=app/models FORCE=y</code> again, and we get:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; bundle exec rake environment elasticsearch:import:all DIR=app/models FORCE=y
[IMPORT] Loading models from: app/models
[IMPORT] Processing model: Rubygem...
[IMPORT] Done
</code></pre></div></div>
<p>Woot!</p>
<p>Lets try the tests.</p>
<p>I’ll start <code class="highlighter-rouge">memcached</code>. Since it just hangs, filling up a terminal tab, lets <a href="https://stackoverflow.com/questions/13338870/what-does-at-the-end-of-a-linux-command-mean">send it to the background</a>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; memcached &amp;
</code></pre></div></div>
<p>and:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bundle exec rake
</code></pre></div></div>
<p>Sweet. Just rows upon rows of sweet green dots…. and a failure.</p>
<p><img src="/images/2018-11-29 at 8.50 PM.png" alt="so close" /></p>
<h2 id="dependencytest">DependencyTest</h2>
<p>The error is:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Failure:
DependencyTest#test_:
with a Gem::Dependency that refers to a Rubygem that exists and has multiple requirements should create a Dependency referring to the existing Rubygem.
[/Users/joshthompson/workspace/rails_speed/rubygems.org/test/unit/dependency_test.rb:96]:
Expected: "&lt; 1.0.0, &gt;= 0.0.0"
Actual: "&gt;= 0.0.0, &lt; 1.0.0"
</code></pre></div></div>
<p>We’ll table this for now.</p>
<p>Starting the server works fine. I’ll update this post if I sort out the test failure.</p>
<p>If you elect to work through the course, this might save you a bit of hassle. I went way too far down the docker hole before just brew installing it. :(</p>
<h2 id="download-rubygems-production-db">download Rubygem’s production DB:</h2>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ script/load-pg-dump -c -d gemcutter_production latest_dump
</code></pre></div></div>
<p>No problems.</p>
<p>When it came time to generate the assets, first take failed:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; RAILS_ENV=production rake assets:precompile
rake aborted!
ArgumentError: Missing `secret_key_base` for 'production' environment, set this string with `rails credentials:edit`
</code></pre></div></div>
<p>Swap it to <code class="highlighter-rouge">&gt; RAILS_ENV=production rake assets:precompile SECRET_KEY_BASE=foo</code> and you’re good to go.</p>
<p>Aaaand…</p>
<p>server error on http://localhost:3000/</p>
<p><img src="/images/2018-11-29 at 9.17 PM.jpg" alt="but what a cute error message" /></p>
<p>A message in the slack group suggested using the following for <code class="highlighter-rouge">script/load-pg-dump</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>script/load-pg-dump -c -d rubygems_production latest_dump
</code></pre></div></div>
<p>(it’s <code class="highlighter-rouge">rubygems_production</code> instead of <code class="highlighter-rouge">gemcutter_production</code>)</p>
<p>still failing. same server error.</p>
<p>Taking a look in <code class="highlighter-rouge">log/production.log</code>, I’ve got useful errors like:</p>
<blockquote>
<p>F, [2018-11-29T21:30:19.469620 #4540] FATAL – : ActionView::Template::Error (PG::UndefinedTable: ERROR: relation “announcements” does not exist</p>
</blockquote>
<p>So:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; RAILS_ENV=production rake db:migrate SECRET_KEY_BASE=foo
</code></pre></div></div>
<p>but that doesn’t work:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; RAILS_ENV=production rake db:migrate SECRET_KEY_BASE=foo
== 20090527122639 CreateRubygems: migrating ===================================
-- adapter_name()
-&gt; 0.0000s
-- adapter_name()
-&gt; 0.0000s
-- adapter_name()
-&gt; 0.0000s
-- create_table(:rubygems, {:id=&gt;:integer})
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::DuplicateTable: ERROR: relation "rubygems" already exists
: CREATE TABLE "rubygems" ("id" serial NOT NULL PRIMARY KEY, "name" character varying, "token" character varying, "user_id" integer, "created_at" timestamp, "updated_at" timestamp)
</code></pre></div></div>
<p>This is probably a horrible fix, but:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ psql
# enter psql cli
\l
# list available databases
\c rubygems_production
# connect to said db
\dt
# list all relations
drop table rubygems;
# drop the table
</code></pre></div></div>
<p>re-run migrations and… it now fails on a different relation.</p>
<p>so, running the mother of all horrible commands:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; RAILS_ENV=production rake db:reset SECRET_KEY_BASE=foo DISABLE_DATABASE_ENVIRONMENT_CHECK=1
</code></pre></div></div>
<p>nukes the production DB, redoes everything. Don’t do this.</p>
<p><code class="highlighter-rouge">RAILS_ENV=production SECRET_KEY_BASE=foo rails s</code></p>
<p>works, now, but the production DB is empty, so we have to re-import it:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>script/load-pg-dump -c -d rubygems_production latest_dump
</code></pre></div></div>
<p>after a few minutes… boot up the rails server again, and:</p>
<p>damnit. error page. Same complaint about missing relation.</p>
<p>When I swap everything out to working on <code class="highlighter-rouge">development</code> environment instead of <code class="highlighter-rouge">production</code>, it works. I’ll carry on from here for now.</p>Josh Thompsonthompsonjoshd@gmail.comYou know the feeling. You are excited to start a guide or a tutorial. You buy it, crack it open, and start working through the environment setup. Then… something goes wrong. Next thing you know, you’ve spent two three too many hours debugging random crap, and you’re not even done with the introduction to the dang thing. Oh, this has never happened to you? Must be nice. I’m working through The Complete Guide to Rails Performance, and I’m thrilled to get learning underway. I’ve hit a few hiccups, though. I’m not the most sophisticated user out there, so here’s a mess of problems I ran into, and the solutions I did. Elastic Search All of my struggles were with ElasticSearch. I’ve never used it before, so this isn’t surprising. The RubyGems docs recommend the following: Install Elastic Search: Pull ElasticSearch 5.1.2 : docker pull docker.elastic.co/elasticsearch/elasticsearch:5.1.2 Running Elasticsearch from the command line: docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" docker.elastic.co/elasticsearch/elasticsearch:5.1.2Growing in your first software development job2018-11-28T13:00:00+00:002018-11-28T13:00:00+00:00https://josh.works/how_to_grow_in_first_dev_role<p>I started my first software developer role a year ago. (November 2017)</p>
<p>This is tremendously exciting, of course, but introduces its own set of challenges, like:</p>
<blockquote>
<p>I finished Turing and I’ve got a job! Oh snap. I just finished a grueling program, and my reward is I’m fit to sit at the same table with people who <em>know so much more than me</em>.</p>
</blockquote>
<blockquote>
<p>I have to learn a whole new codebase. Or four.</p>
</blockquote>
<blockquote>
<p>I have to meet the standards of an inexperienced-but-competent professional, which are higher than “eager student”.</p>
</blockquote>
<p>So, I’ve been thinking <em>a lot</em> about how to grow as a developer in my job.</p>
<h2 id="what-does-grow-as-a-developer-mean">What does “grow as a developer” mean?</h2>
<p>Part of growing in anything means having goals. I have career goals that may be 5+ years out, but for the next year or two, these are the goals I’m using to drive me towards my goal of being “an experienced developer””</p>
<ul>
<li>Build <em>deep</em> competency with the “standard” Rails framework. I think Rails scales just fine, and I don’t want to toss a JS-heavy front-end on our application. (I’m <del>secretly</del> pulling for Turbolinks and Stimulus…)</li>
<li>Help guide our product through a few big changes in the coming year, <em>and help drive decisions around those changes</em></li>
<li>Support other teams within the company</li>
<li>Put up a lot of PRs, close a lot of tickets. Do a lot of “work”</li>
<li>???</li>
</ul>
<p>These are not great goals, as far as good goal-setting goes, but I am not worrying too much about it, because good habits tend to be more important than good goals. And I’ve got good habits!</p>
<h2 id="constraints">Constraints</h2>
<p>Here are some constraints I’m operating within:</p>
<ul>
<li>I work on a 100% distributed team of developers, but most of the rest of the offices works in-office in Pittsburgh. (Project/Product Management, Quality Assurance, DevOps, Support, Managed Services, Sales)</li>
<li>My immediate team is a group of three other developers. Myself and one other dev started at the same time.</li>
<li>The other two developers have been on the team for about three years.</li>
<li>The individuals who built most of the application is no longer on the team.</li>
<li>there is an extreme level of trust passed to all on the team. No one is required to pair with me, or checks in with me every day to make sure I’ve committed X code or fixed Y bugs.</li>
<li>We’re currently without an immediate manager.</li>
<li>I live a balanced life. I spend time with my wife, I rock climb a lot, I’ve gotten back into running, I read a lot. I won’t spend 20 hours a week on top of my job, trying to become a better developer.</li>
</ul>
<!--more-->
<p>These constraints are challenges, indeed. In some ways, it might be easier to waltz into a company that has a robust mentoring program, and batch-hires new developers so everyone is passing through the mentorship pipeline with a buddy.</p>
<p>Sure, that would be cool. But then I’d miss out on all the fun of wrestling with this challenge myself. I’ve rarely found myself in the exact same place, moving in the exact same speed, as many others, so this is a very Josh-friendly opportunity.</p>
<h2 id="what-is-the-range-of-options-available-to-me">What is the range of options available to me?</h2>
<p>There’s a few potential avenues I’m exploring right now:</p>
<ul>
<li>Pairing more with my coworkers</li>
<li>self-study</li>
<li>implement suggestions from other developers</li>
<li>Work-related side-projects</li>
<li>non-work-related side-projects</li>
<li>Meetups</li>
<li>Teaching/helping others</li>
</ul>
<h3 id="pairing-more-with-coworkers">Pairing more with coworkers</h3>
<p>A simplistic solution could be to go look at some company who has an established mentorship program, and try to force-fit it into our team of four. If said plan called for 15 hours a week of pairing, I could ask each of the three developers how they felt about pairing with me for five hours a week, each.</p>
<p>That’s almost a workday out of each week that would be lost to pairing. Maybe this is the most effective use of everyone’s time, but I’m not yet convinced that it is.</p>
<p>If it <em>is</em> the best use of time, there’s still a lot of questions about how it would work, like:</p>
<ul>
<li>how should we pair?</li>
<li>Driver/Navigator? One types, the other adds insights and suggestions?</li>
<li>Do we use my machine or theirs?</li>
<li>Do we work on whatever ticket I’m working on? Whatever ticket they’re working on? both? neither?</li>
</ul>
<p>We’re also 100% remote, so this pairing would have to be 100% Slack/Zoom/something similar. Anytime the internet dropped many packets, there would be lag and friction.</p>
<p>Since I’ve started, I have set up weekly 1:1s with everyone on my team. Usually technical things come up, and I get help with whatever I’m working on, but a big part of it was I wanted to get to know my team. We don’t see each other often, and I wanted to get to know them. I also sometimes “shadow” one of my coworkers for 30 minutes or an hour a week, to absorb some of how they work, without interrupting their work too much.</p>
<h3 id="self-study">Self-study</h3>
<p>There’s a lot of great books out there about software development, and classes, videos, and more.</p>
<p>So far, I’ve read (or am reading)</p>
<ul>
<li><a href="https://pragprog.com/book/ppmetr2/metaprogramming-ruby-2">Metaprogramming in Ruby</a></li>
<li><a href="https://www.poodr.com/">Practical Object-Oriented Design in Ruby</a></li>
<li><a href="https://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X">The Pragmatic Programmer</a></li>
<li><a href="https://pragprog.com/book/cfcar2/the-passionate-programmer">The Nature of Software Development (Ron Jeffries)</a></li>
<li><a href="https://www.goodreads.com/book/show/29902035-agile-web-development-with-rails-5?ac=1&amp;from_search=true">Agile Web Development with Rails 5</a></li>
<li><a href="https://www.goodreads.com/book/show/9364729-eloquent-ruby?ac=1&amp;from_search=true#">Eloquent Ruby</a></li>
<li><a href="https://www.amazon.com/SQL-Queries-Mere-Mortals-Hands/dp/0321992474">SQL Queries for Mere Mortals</a></li>
</ul>
<p>That’ a start, but only that - a start. I don’t yet feel like I’ve read that many books about software development, Ruby/Rails, or programming in general. So, since I love to read, I’ll always have a book or two going on the topic.</p>
<h3 id="courses">Courses</h3>
<p>I done most of Avdi Grimm’s <a href="https://avdi.codes/moom/">Mastering the Object Oriented Mindset</a></p>
<p>It’s been good so far. I’m learning plenty, and I’m about half-way through it, and have been <a href="https://josh.works/tags#moom">writing about some of what I’ve learned</a></p>
<p>I’ve also started Nate Berkopec’s <a href="https://www.railsspeed.com/">RailsSpeed course</a> - it’s all about speeding up a Rails app, and I’m really excited about it.</p>
<p>Both of these have given me a lot of value. In different ways, but I’m feeling like a wiser, more confident, more competent developer because of it.</p>
<p>So far, though, I’ve not been able to take something directly from one of these courses and apply it at Wombat. I’m not concerned about that at all - I’ve not finished either course, and I’ll have plenty of opportunity to apply this learning.</p>
<h3 id="work-related-side-projects">Work-related side-projects</h3>
<p>As I encounter projects at work that are amenable to studying in more depth, and spinning into a one-off tutorial/blog post, I will capture the idea and consider a few hours of effort in that direction.</p>
<p>That is where a few recent blog posts have been born:</p>
<ul>
<li><a href="/pry-tips-and-tricks">Pry Tips and Tricks</a></li>
<li><a href="/sidekiq-and-background-jobs-in-rails-for-beginners">Sidekiq/Redis/Background jobs</a></li>
<li><a href="/array-divergence">array intersection/divergence</a></li>
<li><a href="/test-rake-tasks-in-rails">Testing Rake Tasks</a></li>
</ul>
<p>These blog posts are a great way for me to build a more robust mental model around the topic, as well as give me something to refer back to if I need to later.</p>
<p>This is a nice “proof of progress”, as well. It’s easy for me to feel like I’ve not made much progress in the last week, and therefore <em>I must have made no progress in the last six months</em>. This is a logical fallacy, but that’s how my brain works. It’s nice to have evidence of work done.</p>
<h3 id="non-work-related-side-projects">non-work-related side-projects</h3>
<p>I don’t spend too much time on non-work related projects. I’d love to extend my Turing capstone project, <a href="/block-value">Block Value</a>.</p>
<p>I had a ton of fun with it, and it feels close to being <em>actually good</em>. I’d need to brush up on my Javascript a bit, though. When I’m ready to dig into Javascript, I think this project might be the way I do that. I’d like to drop in OpenStreetMaps instead of Google Maps, and drop in <a href="https://leafletjs.com/">Leaflet.js</a> instead of the crappy Javascript I have in there.</p>
<h3 id="meetups">Meetups</h3>
<p>I’ve been attending <a href="https://www.meetup.com/boulder_ruby_group/">Boulder.rb</a> and <a href="https://www.meetup.com/Denver-rb/">Denver.rb</a> (and am considering starting a meetup in Golden!), and I’ve enjoyed those communities. Both are fairly time consuming, as I have to travel to the location mid-day to dodge rush hour, etc.</p>
<p>I’m giving a talk at Boulder.rb in a few weeks, and hope to keep that trend rolling.</p>
<h3 id="teachinghelping-others">Teaching/helping others</h3>
<p>I like helping others. In job hunts, salary negotiations, learning programming, rock climbing, and more.</p>
<p>I’ve worked regularly with folks breaking into software development (Turing and non-Turing students) and I enjoy the process. It helps me stretch my empathy/teaching/understanding muscles, and I very much enjoy teaching, and seeing others make progress, so this is probably something I’ll continue for the rest of my career.</p>
<p>I know that helping others is a part of growing in a career, but I’m not yet seeing where this interest of mine ties into helping my own skills grow.</p>
<p>As our team at Wombat grows, I’ll get to step into a bit of a mentorship role there, but this opportunity brings this post full circle. I’m not yet exactly sure what I have to offer. Sure, I’ll be useful in them getting their environment set up, and wrapping their head around the code base, and making good PRs and stuff… but I’ve got a lot more I need to learn.</p>
<h3 id="additional-resources">Additional Resources</h3>
<p>I’ve read all of this, and more, in pursuit of upping my skills.</p>
<ul>
<li><a href="https://www.nateliason.com/blog/learning-plateau">How to Break Through Any Learning Plateau and Never Stop Growing (Nat Ellison)</a></li>
<li><a href="https://www.linkedin.com/pulse/want-happy-work-spend-time-learning-josh-bersin/">New Research Shows “Heavy Learners”​ More Confident, Successful, and Happy at Work (LinkedIn)</a></li>
<li><a href="https://slack.engineering/how-slack-supports-junior-engineers-89f6dcfe74a1">How Slack Supports Junior Engineers (Slack)</a></li>
<li><a href="https://www.eventbrite.com/engineering/how-to-support-junior-engineers/">How Your Company Can Support Junior Engineers</a></li>
<li><a href="https://jvns.ca/blog/senior-engineer/">What’s a senior engineer’s job? (Julia Evans)</a></li>
<li><a href="https://www.kitchensoap.com/2012/10/25/on-being-a-senior-engineer/">On Being A Senior Engineer (John Allspaw)</a></li>
<li><a href="https://lethain.com/career-narratives/">Career narratives. (Will Larson)</a></li>
<li><a href="https://news.ycombinator.com/item?id=18128477&amp;p=2">Ask HN: What is your best advice for a junior software developer?</a></li>
</ul>Josh Thompsonthompsonjoshd@gmail.comI started my first software developer role a year ago. (November 2017) This is tremendously exciting, of course, but introduces its own set of challenges, like: I finished Turing and I’ve got a job! Oh snap. I just finished a grueling program, and my reward is I’m fit to sit at the same table with people who know so much more than me. I have to learn a whole new codebase. Or four. I have to meet the standards of an inexperienced-but-competent professional, which are higher than “eager student”. So, I’ve been thinking a lot about how to grow as a developer in my job. What does “grow as a developer” mean? Part of growing in anything means having goals. I have career goals that may be 5+ years out, but for the next year or two, these are the goals I’m using to drive me towards my goal of being “an experienced developer”” Build deep competency with the “standard” Rails framework. I think Rails scales just fine, and I don’t want to toss a JS-heavy front-end on our application. (I’m secretly pulling for Turbolinks and Stimulus…) Help guide our product through a few big changes in the coming year, and help drive decisions around those changes Support other teams within the company Put up a lot of PRs, close a lot of tickets. Do a lot of “work” ??? These are not great goals, as far as good goal-setting goes, but I am not worrying too much about it, because good habits tend to be more important than good goals. And I’ve got good habits! Constraints Here are some constraints I’m operating within: I work on a 100% distributed team of developers, but most of the rest of the offices works in-office in Pittsburgh. (Project/Product Management, Quality Assurance, DevOps, Support, Managed Services, Sales) My immediate team is a group of three other developers. Myself and one other dev started at the same time. The other two developers have been on the team for about three years. The individuals who built most of the application is no longer on the team. there is an extreme level of trust passed to all on the team. No one is required to pair with me, or checks in with me every day to make sure I’ve committed X code or fixed Y bugs. We’re currently without an immediate manager. I live a balanced life. I spend time with my wife, I rock climb a lot, I’ve gotten back into running, I read a lot. I won’t spend 20 hours a week on top of my job, trying to become a better developer.Primitive Obsession &amp; Exceptional Values2018-10-12T13:00:00+00:002018-10-12T13:00:00+00:00https://josh.works/primitive_obsession_and_exceptional_values<p>I’ve been working through Avdi Grimes’ <a href="https://avdi.codes/moom/">Mastering the Object Oriented Mindset</a> course.</p>
<p>One of the topics was using “whole values”, instead of being “primative obsessed”. The example Avdi gave was clear as day.</p>
<p>He used a course with a <code class="highlighter-rouge">duration</code> attribute to show the problem.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">course</span><span class="p">.</span><span class="nf">duration</span>
<span class="o">=&gt;</span> <span class="mi">3</span>
</code></pre></div></div>
<p>3 what? weeks? days? months?</p>
<p>Of course, you could write a method like:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">course</span><span class="p">.</span><span class="nf">duration_in_weeks</span>
<span class="o">=&gt;</span> <span class="mi">3</span>
</code></pre></div></div>
<p>But now you’ll have trouble rendering this all over the place. You’d have conditionals every time you wanted to render courses in weeks (if it makes sense), or in months (if appropriate), or of course, days.</p>
<p>So, the solution is to use “Whole values”. This means an attribute should be a complete unit, in and of itself, and should need no further refining to be usable.</p>
<p>So, you should be able to do something like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">course</span><span class="p">.</span><span class="nf">duration</span>
<span class="o">=&gt;</span> <span class="no">Months</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span>
<span class="n">shorter_course</span><span class="p">.</span><span class="nf">duration</span>
<span class="o">=&gt;</span> <span class="no">Weeks</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span>
</code></pre></div></div>
<!--more-->
<p><img src="https://media.giphy.com/media/26ufdipQqU2lhNA4g/giphy.gif" alt="mind blown" /></p>
<p>So, here’s the basics of this <code class="highlighter-rouge">Duration</code> class, that your units (like <code class="highlighter-rouge">Days</code>, <code class="highlighter-rouge">Weeks</code>, and <code class="highlighter-rouge">Months</code>) inherit from:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Duration</span>
<span class="nb">attr_reader</span> <span class="ss">:magnitude</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">magnitude</span><span class="p">)</span>
<span class="vi">@magnitude</span> <span class="o">=</span> <span class="n">magnitude</span>
<span class="nb">freeze</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">inspect</span>
<span class="s2">"</span><span class="si">#{</span><span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="si">}</span><span class="s2">[</span><span class="si">#{</span><span class="n">magnitude</span><span class="si">}</span><span class="s2">]"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">to_s</span>
<span class="s2">"</span><span class="si">#{</span><span class="n">magnitude</span><span class="si">}</span><span class="s2"> </span><span class="si">#{</span><span class="nb">self</span><span class="p">.</span><span class="nf">class</span><span class="p">.</span><span class="nf">name</span><span class="p">.</span><span class="nf">downcase</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="kp">alias_method</span> <span class="ss">:to_i</span><span class="p">,</span> <span class="ss">:magnitude</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">Days</span> <span class="o">&lt;</span> <span class="no">Duration</span><span class="p">;</span> <span class="k">end</span>
<span class="k">class</span> <span class="nc">Weeks</span> <span class="o">&lt;</span> <span class="no">Duration</span><span class="p">;</span> <span class="k">end</span>
<span class="k">class</span> <span class="nc">Months</span> <span class="o">&lt;</span> <span class="no">Duration</span><span class="p">;</span> <span class="k">end</span>
</code></pre></div></div>
<p>And it delivers pretty cool stuff:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>main:0&gt; Days.new(3)
=&gt; Days[3]
main:0&gt; Days.new(3).to_s
=&gt; "3 days"
main:0&gt; length = Weeks.new(3)
=&gt; Weeks[3]
</code></pre></div></div>
<p>But having the option to call <code class="highlighter-rouge">course.duration</code> and get <code class="highlighter-rouge">Weeks[3]</code> as a response is… amazing. Or <code class="highlighter-rouge">course.length.to_s</code> and get <code class="highlighter-rouge">3 weeks</code>. Super cool.</p>
<p>Avdi walked through the example code, but I was partial to having it available for playing around myself. So, I built a very simple test file.</p>
<p><a href="https://gist.github.com/josh-works/6eb437670b66a67675c23352c787e66d">Check out the full test suite, if you’re interested</a></p>
<p>The above gist also has the code that makes it all pass. I’m going to highlight just a few of the tests below:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_duration_is_months_object</span>
<span class="n">assert_instance_of</span> <span class="no">Months</span><span class="p">,</span> <span class="vi">@math</span><span class="p">.</span><span class="nf">duration</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This test (and a few others) make it explicit that when you call <code class="highlighter-rouge">@math.duration</code> you don’t expect a primitive back - you expect an instance of the <code class="highlighter-rouge">Months</code> class. Super cool.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_duration_inspect</span>
<span class="n">assert_equal</span> <span class="s2">"Months[4]"</span><span class="p">,</span> <span class="vi">@math</span><span class="p">.</span><span class="nf">duration</span><span class="p">.</span><span class="nf">inspect</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We can “convert” our Duration value into a primitive (a string) by calling <code class="highlighter-rouge">#inspect</code> on it. Other than this, though, the duration value lives as its own object.</p>
<p>The tests test some helper methods that Avidi mentioned, to make it a bit easier to render the <code class="highlighter-rouge">course.duration</code> in a view:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">render_course_info</span><span class="p">(</span><span class="n">course</span><span class="p">)</span>
<span class="s2">"</span><span class="si">#{</span><span class="n">course</span><span class="p">.</span><span class="nf">name</span><span class="si">}</span><span class="s2"> (</span><span class="si">#{</span><span class="n">render_value</span><span class="p">(</span><span class="n">course</span><span class="p">.</span><span class="nf">duration</span><span class="p">)</span><span class="si">}</span><span class="s2">)"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">render_value</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">case</span> <span class="n">value</span>
<span class="k">when</span> <span class="no">Months</span>
<span class="s2">"</span><span class="si">#{</span><span class="n">value</span><span class="p">.</span><span class="nf">to_i</span><span class="si">}</span><span class="s2"> gruling months"</span>
<span class="k">when</span> <span class="no">Weeks</span>
<span class="s2">"</span><span class="si">#{</span><span class="n">value</span><span class="p">.</span><span class="nf">to_i</span><span class="si">}</span><span class="s2"> delightful weeks"</span>
<span class="k">when</span> <span class="no">Days</span>
<span class="s2">"a paultry </span><span class="si">#{</span><span class="n">value</span><span class="p">.</span><span class="nf">to_i</span><span class="si">}</span><span class="s2"> days"</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<h1 id="exceptional-values">Exceptional Values</h1>
<p>So, we’ve got this method that takes input as a string, like “12 months”, and tries to convert it to <code class="highlighter-rouge">Months[12]</code>.</p>
<p>If you are accepting data from a user, you’ll need to plan on invalid input, like “99 blinks”.</p>
<p>Here’s the first take of the conversion method:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">Duration</span><span class="p">(</span><span class="n">raw_value</span><span class="p">)</span>
<span class="k">case</span> <span class="n">raw_value</span>
<span class="k">when</span> <span class="no">Duration</span>
<span class="n">raw_value</span>
<span class="k">when</span> <span class="sr">/\A(\d+)\s+months\z/i</span>
<span class="no">Months</span><span class="p">[</span><span class="vg">$1</span><span class="p">.</span><span class="nf">to_i</span><span class="p">]</span>
<span class="k">when</span> <span class="sr">/\A(\d+)\s+weeks\z/i</span>
<span class="no">Weeks</span><span class="p">[</span><span class="vg">$1</span><span class="p">.</span><span class="nf">to_i</span><span class="p">]</span>
<span class="k">when</span> <span class="sr">/\A(\d+)\s+days\z/i</span>
<span class="no">Days</span><span class="p">[</span><span class="vg">$1</span><span class="p">.</span><span class="nf">to_i</span><span class="p">]</span>
<span class="k">else</span>
<span class="kp">nil</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This kind of works, but <code class="highlighter-rouge">nil</code> isn’t a great place-holder. Now your view logic needs to do all sorts of special work to handle if there are <code class="highlighter-rouge">nil</code> values, which of course there will be all the time, because if you call:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">course</span> <span class="o">=</span> <span class="no">Course</span><span class="p">.</span><span class="nf">new</span>
<span class="n">course</span><span class="p">.</span><span class="nf">name</span> <span class="o">=</span> <span class="s2">"math"</span>
<span class="n">course</span><span class="p">.</span><span class="nf">duration</span> <span class="o">=</span> <span class="s2">"12 days"</span>
<span class="n">course</span><span class="p">.</span><span class="nf">save</span>
</code></pre></div></div>
<p>The course <em>will have nil values auto-assigned</em> simply because the user has not filled it in yet.</p>
<p>Anyway, so, as you might expect from someone talking about “whole values”, there’s a “whole value” implementation of an exception:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">Duration</span><span class="p">(</span><span class="n">raw_value</span><span class="p">)</span>
<span class="k">case</span> <span class="n">raw_value</span>
<span class="k">when</span> <span class="no">Duration</span>
<span class="n">raw_value</span>
<span class="k">when</span> <span class="sr">/\A(\d+)\s+months\z/i</span>
<span class="no">Months</span><span class="p">[</span><span class="vg">$1</span><span class="p">.</span><span class="nf">to_i</span><span class="p">]</span>
<span class="k">when</span> <span class="sr">/\A(\d+)\s+weeks\z/i</span>
<span class="no">Weeks</span><span class="p">[</span><span class="vg">$1</span><span class="p">.</span><span class="nf">to_i</span><span class="p">]</span>
<span class="k">when</span> <span class="sr">/\A(\d+)\s+days\z/i</span>
<span class="no">Days</span><span class="p">[</span><span class="vg">$1</span><span class="p">.</span><span class="nf">to_i</span><span class="p">]</span>
<span class="k">else</span>
<span class="no">ExceptionalValue</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">raw_value</span><span class="p">,</span> <span class="ss">reason: </span><span class="s2">"unrecognized format"</span><span class="p">)</span>
<span class="c1"># we create a new Exceptional Value object if we get unrecognized input</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Here’s what that object might look like, using this exceptional value:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">math</span> <span class="o">=</span> <span class="no">Course</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="s2">"Math"</span><span class="p">)</span>
<span class="n">math</span><span class="p">.</span><span class="nf">duration</span> <span class="o">=</span> <span class="s2">"a blink of an eye"</span>
<span class="o">=&gt;</span> <span class="o">&lt;</span><span class="n">struct</span> <span class="no">Course</span>
<span class="nb">name</span><span class="o">=</span><span class="s2">"Math"</span><span class="p">,</span>
<span class="n">duration</span><span class="o">=&lt;</span><span class="no">ExceptionalValue</span><span class="p">:</span><span class="mh">0x00007fca79021188</span> <span class="vi">@raw_value</span><span class="o">=</span><span class="s2">"a blink of an eye"</span><span class="p">,</span> <span class="vi">@reason</span><span class="o">=</span><span class="s2">"unrecognized format"</span><span class="o">&gt;&gt;</span>
</code></pre></div></div>
<p>pretty cool, huh?</p>
<p><a href="https://gist.github.com/josh-works/6eb437670b66a67675c23352c787e66d">check out the gist for tests and class code. Don’t judge me for sticking like 40 classes in the same file…</a></p>Josh Thompsonthompsonjoshd@gmail.comI’ve been working through Avdi Grimes’ Mastering the Object Oriented Mindset course. One of the topics was using “whole values”, instead of being “primative obsessed”. The example Avdi gave was clear as day. He used a course with a duration attribute to show the problem. course.duration =&gt; 3 3 what? weeks? days? months? Of course, you could write a method like: course.duration_in_weeks =&gt; 3 But now you’ll have trouble rendering this all over the place. You’d have conditionals every time you wanted to render courses in weeks (if it makes sense), or in months (if appropriate), or of course, days. So, the solution is to use “Whole values”. This means an attribute should be a complete unit, in and of itself, and should need no further refining to be usable. So, you should be able to do something like this: course.duration =&gt; Months[3] shorter_course.duration =&gt; Weeks[5]`ls` command to show directory contents2018-10-03T11:00:00+00:002018-10-03T11:00:00+00:00https://josh.works/unix_ls_command<p>I like to use the <code class="highlighter-rouge">tree</code> command on my local machine when trying to peek into the structure and contents of a given directory.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>tree -L 2
</code></pre></div></div>
<p>will [L]ist recursively everything [2] levels deep from your current directory. The output is nicely formatted like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; tree -L 2
.
├── cargo
│ ├── ARCHITECTURE.md
│ ├── CONTRIBUTING.md
│ ├── Cargo.lock
│ ├── Cargo.toml
│ ├── LICENSE-APACHE
│ ├── LICENSE-MIT
│ ├── LICENSE-THIRD-PARTY
│ ├── README.md
│ ├── appveyor.yml
│ ├── rustfmt.toml
│ ├── src
│ ├── target
│ └── tests
├── fastly
│ └── fastly-test-blog
├── get-pip.py
├── learning_aws
├── learning_elixir
│ ├── hellow
│ ├── sample01.exs
│ └── sling_clone
├── learning_over_the_wire
│ ├── bandit
│ └── leviathan
├── learning_react
│ └── react_tutorial
├── learning_ruby_rails
│ ├── InstaClone
│ ├── blocks_practice.rb
│ ├── chris_pine_ruby_lessons
│ ├── eloquent_ruby
│ ├── email_sender
</code></pre></div></div>
<p>If you’ve SSHed into a linux box, however, and you’re trying to look around a bit, you won’t have <code class="highlighter-rouge">tree</code> available to you. How can you list out the contents of directories?</p>
<p>Easy. The good ‘ol <code class="highlighter-rouge">ls</code> command:</p>
<!--more-->
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ls -1d ./*/*
</code></pre></div></div>
<p>results in:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; ls -1d ./*/*
./cargo/ARCHITECTURE.md
./cargo/CONTRIBUTING.md
./cargo/Cargo.lock
./cargo/Cargo.toml
./cargo/LICENSE-APACHE
./cargo/LICENSE-MIT
./cargo/LICENSE-THIRD-PARTY
./cargo/README.md
./cargo/appveyor.yml
./cargo/rustfmt.toml
./cargo/src
./cargo/target
./cargo/tests
./fastly/fastly-test-blog
./learning_elixir/hellow
./learning_elixir/sample01.exs
./learning_elixir/sling_clone
./learning_over_the_wire/bandit
./learning_over_the_wire/leviathan
./learning_react/react_tutorial
./learning_ruby_rails/InstaClone
./learning_ruby_rails/blocks_practice.rb
./learning_ruby_rails/chris_pine_ruby_lessons
./learning_ruby_rails/eloquent_ruby
./learning_ruby_rails/email_sender
</code></pre></div></div>
<p><code class="highlighter-rouge">ls</code> is obviously “list”. What about that <code class="highlighter-rouge">-1d</code> thing?</p>
<p>according to the man page:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>-1 (The numeric digit ``one''.) Force output to be one entry per line. This is the
default when output is not to a terminal.
-d Directories are listed as plain files (not searched recursively).
</code></pre></div></div>
<p>to achieve a certain number of directories in depth, just repeat the <code class="highlighter-rouge">./*./*</code> pattern, one <code class="highlighter-rouge">/*</code> per directory depth you’d like to view.</p>
<p>The output isn’t always particularly readable - sometimes you get a ton of results.</p>
<p>If this is the case, you could always pipe the output to <code class="highlighter-rouge">head</code>, or <code class="highlighter-rouge">grep</code> the output. That said, I’ve used this general pattern regularly, and it’s been helpful.</p>
<h3 id="related-resources">Related resources</h3>
<ul>
<li><a href="https://unix.stackexchange.com/questions/93323/list-subdirectories-only-n-level-deep">List subdirectories only n level deep (unix.stackexchange.com)</a></li>
</ul>Josh Thompsonthompsonjoshd@gmail.comI like to use the tree command on my local machine when trying to peek into the structure and contents of a given directory. tree -L 2 will [L]ist recursively everything [2] levels deep from your current directory. The output is nicely formatted like this: &gt; tree -L 2 . ├── cargo │ ├── ARCHITECTURE.md │ ├── CONTRIBUTING.md │ ├── Cargo.lock │ ├── Cargo.toml │ ├── LICENSE-APACHE │ ├── LICENSE-MIT │ ├── LICENSE-THIRD-PARTY │ ├── README.md │ ├── appveyor.yml │ ├── rustfmt.toml │ ├── src │ ├── target │ └── tests ├── fastly │ └── fastly-test-blog ├── get-pip.py ├── learning_aws ├── learning_elixir │ ├── hellow │ ├── sample01.exs │ └── sling_clone ├── learning_over_the_wire │ ├── bandit │ └── leviathan ├── learning_react │ └── react_tutorial ├── learning_ruby_rails │ ├── InstaClone │ ├── blocks_practice.rb │ ├── chris_pine_ruby_lessons │ ├── eloquent_ruby │ ├── email_sender If you’ve SSHed into a linux box, however, and you’re trying to look around a bit, you won’t have tree available to you. How can you list out the contents of directories? Easy. The good ‘ol ls command:Rails Migration: When you can’t add a uniqueness constraint because you already have duplicates2018-09-28T10:00:00+00:002018-09-28T10:00:00+00:00https://josh.works/rails_migrations_add_unique_constraint_with_existing_duplicates<p><em>I get to occasionally contribute to the Wombat Security dev blog. I wrote the following for <a href="http://development.wombatsecurity.com/development/2018/09/28/rails-migration-add-uniqueness-constraint/">development.wombatsecurity.com</a>.</em></p>
<hr />
<p>For work, I picked up a bug where a CSV export was creating duplicate rows when it shouldn’t have been.</p>
<p>We generate the CSV export with a big ol’ SQL statement in some Elixir workers, and got the bug reproduced and found the problem was with how we did some joins on other columns.</p>
<p>We had something in the statement like this:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">`reports`</span> <span class="k">ON</span> <span class="nv">`reports`</span><span class="p">.</span><span class="nv">`id`</span> <span class="o">=</span> <span class="nv">`people`</span><span class="p">.</span><span class="nv">`report_id`</span>
<span class="k">LEFT</span> <span class="k">JOIN</span> <span class="nv">`addresses`</span> <span class="k">ON</span> <span class="nv">`addresses`</span><span class="p">.</span><span class="nv">`people_id`</span> <span class="o">=</span> <span class="nv">`people`</span><span class="p">.</span><span class="nv">`id`</span>
<span class="k">LEFT</span> <span class="k">JOIN</span> <span class="nv">`favorite_colors`</span> <span class="k">ON</span> <span class="nv">`favorite_colors`</span><span class="p">.</span><span class="nv">`people_id`</span> <span class="o">=</span> <span class="nv">`people.id`</span>
</code></pre></div></div>
<p>In this highly contrived example, I found out that we’re expecting a single row in the <code class="highlighter-rouge">favorite_colors</code> table for each <code class="highlighter-rouge">person</code>, but we were getting multiple <code class="highlighter-rouge">favorite_color</code> rows.</p>
<p>Every time we had a duplicate row on that joined table, the <code class="highlighter-rouge">LEFT JOIN</code> created two rows in the export, even though there should have been one.</p>
<!--more-->
<p><a href="https://robots.thoughtbot.com/the-perils-of-uniqueness-validations">Thoughtbot has an amazing article about the exact problem we were having</a>.</p>
<p>In that article, they describe the problem with uniqueness validations like so:</p>
<p><img src="/images/unique_without_index.png" alt="unique_without_index" /></p>
<p><em><a href="https://robots.thoughtbot.com/the-perils-of-uniqueness-validations">https://robots.thoughtbot.com/the-perils-of-uniqueness-validations</a></em></p>
<p>We need to enforce uniqueness at the <em>database</em> level. Not the model level.</p>
<h2 id="whats-the-easy-fix">What’s the easy fix?</h2>
<p>The “easy” fix is simple - Add an index to the column you want to enforce uniqueness upon, then add a uniqueness constraint:</p>
<p>(since this column already happened to have an index in our database, we have to remove it and re-add it in the migration)</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">AddUniquenessConstraintToFavoriteColors</span> <span class="o">&lt;</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Migration</span>
<span class="k">def</span> <span class="nf">change</span>
<span class="n">remove_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span>
<span class="n">add_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span><span class="p">,</span> <span class="ss">unique: </span><span class="kp">true</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Lets make this a bit more explicit. We’ll define an <code class="highlighter-rouge">up</code> and <code class="highlighter-rouge">down</code> migration. (you’ll see why in a moment)</p>
<h3 id="updown-migration">Up/down migration</h3>
<p>I am going to be explicit about the up and down, for the rollback:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">AddUniquenessConstraintToFavoriteColors</span> <span class="o">&lt;</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Migration</span>
<span class="k">def</span> <span class="nf">up</span>
<span class="n">remove_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span>
<span class="c1"># you can't MODIFY the index, just remove it, then re-add it with the changes</span>
<span class="n">add_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span><span class="p">,</span> <span class="ss">unique: </span><span class="kp">true</span><span class="p">,</span> <span class="ss">algorithm: :inplace</span>
<span class="c1"># unique forces uniqueness (duh)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">down</span>
<span class="n">remove_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span>
<span class="c1"># removing the index that has uniqueness constraint</span>
<span class="n">add_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span>
<span class="c1"># adding one without the constraint</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This migration will work, <em>if</em> you run it against a table that doesn’t have duplicate values in the given column.</p>
<p>But the <em>whole reason</em> I was digging into all this was because I had a table with duplicate values. This is a table that now has many rows of data, and more than a few duplicates.</p>
<p>If you run the above migration, and there are duplicates on the table, you’ll get a lovely error like so:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>== 20180914203948 AddUniquenessConstraintToFavoriteColors: migrating ========================
-- remove_index(:favorite_colors, :person_id)
-&gt; 0.0388s
-- add_index(:favorite_colors, :person_id, {:unique=&gt;true, :algorithm=&gt;:inplace})
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Mysql2::Error: Duplicate entry '44' for key 'index_favorite_colors_on_person_id': CREATE UNIQUE INDEX `index_favorite_colors_on_person_id` ON `favorite_colors` (`person_id`) ALGORITHM = INPLACE
</code></pre></div></div>
<h1 id="clean-up-duplicates-and-add-uniqueness-constraint">Clean up duplicates <em>and</em> add uniqueness constraint</h1>
<p>The big question was:</p>
<blockquote>
<p>How do we clean up the duplicates, <em>and</em> run the migration, without duplicates showing up between when we clean them up and run the migration?</p>
</blockquote>
<p>My first thought was a rake task to find and prune duplicates, but this is a relatively active table, and if even a few seconds elapsed between the rake task and migration, we might get another duplicate on it, which would prevent the migration from running.</p>
<h1 id="the-solution">The Solution</h1>
<p>The rake task was not a good solution. Fortunately, I work with many people who are much smarter than I, and they put me on the right track. I needed to update the migration to run a query that would:</p>
<ol>
<li>Find all duplicate rows</li>
<li>Get the ID’s of those duplicate rows</li>
<li>Trim one of the row IDs of the “duplicates” list</li>
<li>Delete all the remaining IDs.</li>
</ol>
<p>So, the SQL query gets built up in three pieces.</p>
<p>I’ll give you the full SQL query, then we’ll dig into the component pieces. The following query creates a list of IDs that we can then use ActiveRecord/Sequel to delete. (More on the migration in a minute):</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="k">substring</span><span class="p">(</span><span class="n">dups</span><span class="p">.</span><span class="n">id_groups</span><span class="p">,</span> <span class="k">position</span><span class="p">(</span><span class="s1">','</span><span class="k">IN</span> <span class="n">dups</span><span class="p">.</span><span class="n">id_groups</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">FROM</span>
<span class="p">(</span><span class="k">SELECT</span> <span class="n">group_concat</span><span class="p">(</span><span class="n">fc</span><span class="p">.</span><span class="n">id</span><span class="p">)</span> <span class="k">AS</span> <span class="n">id_groups</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">fc</span>
<span class="k">WHERE</span> <span class="k">EXISTS</span>
<span class="p">(</span>
<span class="k">SELECT</span> <span class="mi">1</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">tmp</span>
<span class="k">WHERE</span> <span class="n">tmp</span><span class="p">.</span><span class="n">person_id</span> <span class="o">=</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
<span class="k">LIMIT</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span>
<span class="p">)</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
<span class="p">)</span> <span class="k">AS</span> <span class="n">dups</span>
</code></pre></div></div>
<p>Lets unpack it. First, here’s a little SQL you could run locally to generate results to play with:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">favorite_colors</span> <span class="p">(</span>
<span class="n">ID</span> <span class="n">INT</span> <span class="n">AUTO_INCREMENT</span> <span class="k">PRIMARY</span> <span class="k">KEY</span> <span class="p">,</span>
<span class="n">person_id</span> <span class="n">INT</span><span class="p">,</span>
<span class="n">color</span> <span class="n">VARCHAR</span><span class="p">(</span><span class="mi">255</span><span class="p">)</span>
<span class="p">);</span>
<span class="k">INSERT</span> <span class="k">INTO</span> <span class="nv">`favorite_colors`</span> <span class="p">(</span><span class="nv">`ID`</span><span class="p">,</span> <span class="nv">`person_id`</span><span class="p">,</span> <span class="nv">`color`</span><span class="p">)</span>
<span class="k">VALUES</span>
<span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'yellow'</span><span class="p">),</span>
<span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="s1">'yellow'</span><span class="p">),</span>
<span class="p">(</span><span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'blue'</span><span class="p">),</span>
<span class="p">(</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'blue'</span><span class="p">),</span>
<span class="p">(</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'green'</span><span class="p">),</span>
<span class="p">(</span><span class="mi">6</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="s1">'purple'</span><span class="p">),</span>
<span class="p">(</span><span class="mi">7</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="s1">'yellow'</span><span class="p">),</span>
<span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="s1">'black'</span><span class="p">);</span>
</code></pre></div></div>
<p>OK, first, lets find the duplicate rows. There’s a lot of them in here. The following query comes from <a href="https://stackoverflow.com/a/689294/3210178">StackOverflow</a>, and served me well:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">fc</span>
<span class="k">WHERE</span> <span class="k">EXISTS</span>
<span class="p">(</span>
<span class="k">SELECT</span> <span class="mi">1</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">tmp</span>
<span class="k">WHERE</span> <span class="n">tmp</span><span class="p">.</span><span class="n">person_id</span> <span class="o">=</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
<span class="k">LIMIT</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span>
<span class="p">)</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
</code></pre></div></div>
<p>So, obviously, that <code class="highlighter-rouge">WHERE EXISTS</code> piece is interesting. But that depends on the <code class="highlighter-rouge">SELECT 1... LIMIT 1, 1</code>, which is unfamiliar to me.</p>
<h2 id="limit-count-offset">Limit <em>count</em>, <em>offset</em></h2>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="mi">1</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span>
<span class="k">LIMIT</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span>
</code></pre></div></div>
<p>This is the most basic version of the above statement. The <code class="highlighter-rouge">SELECT</code> piece just inserts a <code class="highlighter-rouge">1</code> in the table. It could easily be <code class="highlighter-rouge">SELECT "FOOBAR"</code>. The prior <code class="highlighter-rouge">WHERE EXISTS</code> is just looking for <em>a</em> value from the statement.</p>
<p>The <code class="highlighter-rouge">LIMIT 1, 1</code> is curious. It’s <code class="highlighter-rouge">LIMIT &lt;count&gt; &lt;offset&gt;</code>. So it’s limiting the count to one, and it’s offsetting it by 1. <a href="https://www.postgresql.org/docs/8.2/static/queries-limit.html">PostgreSQL has the best docs on this function</a>.</p>
<p>This means it returns one row, offset from the first row by 1. So, if you ran the following query:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span>
<span class="k">LIMIT</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span>
</code></pre></div></div>
<p>You’d get four rows, <em>skipping</em> the first two rows, and including the subsequent four.</p>
<p>Here’s the results of:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span>
<span class="k">LIMIT</span> <span class="mi">2</span>
</code></pre></div></div>
<table>
<thead>
<tr>
<th>id</th>
<th>person_id</th>
<th>color</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>1</td>
<td>yellow</td>
</tr>
<tr>
<td>2</td>
<td>1</td>
<td>yellow</td>
</tr>
</tbody>
</table>
<p>And if we run</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span>
<span class="k">LIMIT</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span>
</code></pre></div></div>
<p>we get:</p>
<table>
<thead>
<tr>
<th>id</th>
<th>person_id</th>
<th>color</th>
</tr>
</thead>
<tbody>
<tr>
<td>3</td>
<td>2</td>
<td>blue</td>
</tr>
<tr>
<td>4</td>
<td>2</td>
<td>blue</td>
</tr>
</tbody>
</table>
<p>So, <code class="highlighter-rouge">LIMIT 1, 1</code> takes the results of the prior <code class="highlighter-rouge">WHERE</code> and basically “hides” the first match, and leaves only the second match. The temporary result would be empty when the <code class="highlighter-rouge">EXISTS</code> statement checked the subquery, and the row would be deemed “not a duplicate”.</p>
<h4 id="exists"><code class="highlighter-rouge">EXISTS</code></h4>
<p>In <a href="https://dev.mysql.com/doc/refman/8.0/en/exists-and-not-exists-subqueries.html">the docs</a>, we learn:</p>
<blockquote>
<p>If a subquery returns any rows at all, EXISTS subquery is TRUE, and NOT EXISTS subquery is FALSE. For example:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">column1</span> <span class="k">FROM</span> <span class="n">t1</span> <span class="k">WHERE</span> <span class="k">EXISTS</span> <span class="p">(</span><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">t2</span><span class="p">);</span>
</code></pre></div> </div>
</blockquote>
<p>I still don’t fully grasp the nuance of the total SQL statement, but I feel like I’m approaching comprehension. Either way, I am pleased to know how to return the complete rows of duplicates from a table where there are duplicate values in a given column.</p>
<p>OK, so we’ve got duplicates. How do we make them usable?</p>
<h3 id="get-ids-from-results">Get IDs from results</h3>
<p>We don’t want to <code class="highlighter-rouge">SELECT *</code>, now, we want to <code class="highlighter-rouge">SELECT group_concat(favorite_colors.id) AS id_groups</code>.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">group_concat</span><span class="p">(</span><span class="n">fc</span><span class="p">.</span><span class="n">id</span><span class="p">)</span> <span class="k">AS</span> <span class="n">id_groups</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">fc</span>
<span class="k">WHERE</span> <span class="k">EXISTS</span>
<span class="p">(</span>
<span class="k">SELECT</span> <span class="mi">1</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">tmp</span>
<span class="k">WHERE</span> <span class="n">tmp</span><span class="p">.</span><span class="n">person_id</span> <span class="o">=</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
<span class="k">LIMIT</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span>
<span class="p">)</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
</code></pre></div></div>
<p>returns:</p>
<table>
<thead>
<tr>
<th>id_groups</th>
</tr>
</thead>
<tbody>
<tr>
<td>1,2</td>
</tr>
<tr>
<td>3,4,6</td>
</tr>
<tr>
<td>5,7</td>
</tr>
</tbody>
</table>
<p>The <code class="highlighter-rouge">GROUP BY table.column_with_duplicates</code> is important to split the groups by unique value. Without the group by, you’d get one long string of joined IDs, which would be totally useless.</p>
<p>Next, we want to trim off the first ID in each of these groups. So, here’s the full query:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="k">substring</span><span class="p">(</span><span class="n">dups</span><span class="p">.</span><span class="n">id_groups</span><span class="p">,</span> <span class="k">position</span><span class="p">(</span><span class="s1">','</span><span class="k">IN</span> <span class="n">dups</span><span class="p">.</span><span class="n">id_groups</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="k">FROM</span> <span class="p">(</span>
<span class="k">SELECT</span> <span class="n">group_concat</span><span class="p">(</span><span class="n">fc</span><span class="p">.</span><span class="n">id</span><span class="p">)</span> <span class="k">AS</span> <span class="n">id_groups</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">fc</span>
<span class="k">WHERE</span> <span class="k">EXISTS</span>
<span class="p">(</span>
<span class="k">SELECT</span> <span class="mi">1</span>
<span class="k">FROM</span> <span class="n">favorite_colors</span> <span class="n">tmp</span>
<span class="k">WHERE</span> <span class="n">tmp</span><span class="p">.</span><span class="n">person_id</span> <span class="o">=</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
<span class="k">LIMIT</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span>
<span class="p">)</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">fc</span><span class="p">.</span><span class="n">person_id</span>
<span class="p">)</span> <span class="k">AS</span> <span class="n">dups</span>
</code></pre></div></div>
<p>And this returns:</p>
<table>
<thead>
<tr>
<th>id_groups</th>
</tr>
</thead>
<tbody>
<tr>
<td>2</td>
</tr>
<tr>
<td>4,6</td>
</tr>
<tr>
<td>7</td>
</tr>
</tbody>
</table>
<p>And the results of this SQL statement can now be deleted from the database via ActiveRecord. Here’s our full migration:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">AddUniquenessConstraintToFavoriteColors</span> <span class="o">&lt;</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Migration</span>
<span class="n">disable_ddl_transaction!</span>
<span class="k">def</span> <span class="nf">up</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">execute</span> <span class="o">&lt;&lt;-</span><span class="no">SQL</span><span class="sh">
SELECT substring(dups.id_groups, position(','IN dups.id_groups) + 1)
FROM
(SELECT group_concat(fc.id) AS id_groups
FROM favorite_colors fc
WHERE EXISTS
(
SELECT 1
FROM favorite_colors tmp
WHERE tmp.person_id = fc.person_id
LIMIT 1, 1
)
GROUP BY fc.person_id
) AS dups
</span><span class="no"> SQL</span>
<span class="n">results</span><span class="p">.</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">id_array</span><span class="o">|</span>
<span class="c1"># I know find_in_batches would normally be a better fit</span>
<span class="no">FavoriteColors</span><span class="p">.</span><span class="nf">where</span><span class="p">(</span><span class="ss">id: </span><span class="n">id_array</span><span class="p">).</span><span class="nf">delete_all</span>
<span class="k">end</span>
<span class="n">remove_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span>
<span class="n">add_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span><span class="p">,</span> <span class="ss">unique: </span><span class="kp">true</span><span class="p">,</span> <span class="ss">algorithm: :inplace</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">down</span>
<span class="n">remove_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span>
<span class="n">add_index</span> <span class="ss">:favorite_colors</span><span class="p">,</span> <span class="ss">:person_id</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<h1 id="in-conclusion">In Conclusion</h1>
<p>You can use this pattern to add a uniqueness constraint to a table that already has duplicate values. This will clean out duplicates, but leave original values, and will prevent additional duplicates from being written to the table.</p>
<p>In hindsight, this was a relatively straight-forward migration. I had not found any resource online that talked about the process of adding a uniqueness constraint if the table already had data that violated the constraint, so I hope that this write-up might help someone else in a similar spot.</p>
<h3 id="useful-additional-resources">Useful additional resources</h3>
<ul>
<li><a href="https://robots.thoughtbot.com/the-perils-of-uniqueness-validations">The Perils of Uniqueness Validations (Thoughtbot)</a></li>
<li><a href="https://stackoverflow.com/questions/48007376/rails-add-unique">Rails add unique (StackOverflow question)</a></li>
<li><a href="https://medium.com/@igorkhomenko/rails-make-sure-you-have-proper-db-indexes-for-your-models-unique-validations-ffd0364df26f">Rails: make sure you have proper DB indexes for your model’s unique validations (Igor Khomenko, Medium)</a></li>
</ul>Josh Thompsonthompsonjoshd@gmail.comI get to occasionally contribute to the Wombat Security dev blog. I wrote the following for development.wombatsecurity.com. For work, I picked up a bug where a CSV export was creating duplicate rows when it shouldn’t have been. We generate the CSV export with a big ol’ SQL statement in some Elixir workers, and got the bug reproduced and found the problem was with how we did some joins on other columns. We had something in the statement like this: LEFT OUTER JOIN `reports` ON `reports`.`id` = `people`.`report_id` LEFT JOIN `addresses` ON `addresses`.`people_id` = `people`.`id` LEFT JOIN `favorite_colors` ON `favorite_colors`.`people_id` = `people.id` In this highly contrived example, I found out that we’re expecting a single row in the favorite_colors table for each person, but we were getting multiple favorite_color rows. Every time we had a duplicate row on that joined table, the LEFT JOIN created two rows in the export, even though there should have been one.Sidekiq and Background Jobs for Beginners2018-07-30T13:00:00+00:002018-07-30T13:00:00+00:00https://josh.works/sidekiq_background_jobs_for_beginners<p>I’ve recently had to learn more about background jobs (using <a href="https://github.com/mperham/sidekiq">Sidekiq</a>, specifically) for some bugs I was working on.</p>
<p>I learned a lot. Much of it was <em>extremely</em> basic. Anyone who knows much at all about Sidekiq will say “oh, duh, of course that’s true”, but at the time, it wasn’t obvious to me.</p>
<p>The reason I needed such basic overviews is because prior to my current job, I’d had just a few <em>hours</em> of exposure to background jobs, and understood little of those hours. And I got dropped into a project that has dozens of jobs, handling hundreds of thousands of actions a day.</p>
<p>As is my style, when I don’t understand something, I like to go to the very basics.</p>
<p>Most of the interesting stuff is way down at the bottom, on <a href="https//josh.works/sidekiq-and-background-jobs-in-rails-for-beginners#watching-redis">watching Redis do it’s thing</a></p>
<!--more-->
<p>In this case, I went back to Turing! I found the <a href="http://backend.turing.io/module3/lessons/intro_to_background_workers">background jobs lesson</a> from Mod 3, and worked through it.</p>
<p>I very much enjoy seeing evidence of things working “under the hood”, rather than just accepting that <code class="highlighter-rouge">BackgroundWorker.perform_later(foo.id)</code> works differently than <code class="highlighter-rouge">BackgroundWorker.new.perform(foo.id)</code>, etc. So, this post will focus not as much on <em>using</em> Sidekiq, but <em>seeing that it’s working</em>.</p>
<p>If you want to follow along, do the above tutorial. <a href="https://github.com/josh-works/turing_sidekiq_tutorial/tree/eb5ef7eb34f8baefab9d763c469d9917c09c7d3f">This is what my repo looks like right now</a>. I’ll recap most of what’s in the tutorial.</p>
<hr />
<p>To run the app, run each of the following, using multiple terminal tabs as needed:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails s
redis-server
sidekiq
mailcatcher
</code></pre></div></div>
<p>The app should now working. Navigate to <code class="highlighter-rouge">http://localhost:3000/</code>, and you should see Missy Elliot in all her glory.</p>
<p>Open up <code class="highlighter-rouge">http://localhost:3000/sidekiq/</code> to see the sidekiq dashboard, and then over to <code class="highlighter-rouge">http://localhost:1080/</code> for mailcatcher.</p>
<p>You’ll notice that when sending emails via the app, nothing is happening on [http://localhost:3000/sidekiq/], and the redis terminal window is untouched:</p>
<p><img src="/images/2018-07-25_redis.jpg" alt="redis" /></p>
<h1 id="convert-a-non-background-job-to-a-background-job">Convert a non-background-job to a background job</h1>
<p>The essence of a background job is to do stuff <em>in the background</em>, without making the Rails app sit around doing all the work.</p>
<p>To simulate the pain of waiting for synchronous jobs, when you use this app, the “send email” method has a five-second <code class="highlighter-rouge">sleep</code> in it.</p>
<p>Lets make this a background job:</p>
<ol>
<li>create the job. (you can use <code class="highlighter-rouge">rails generate job &lt;job_name&gt;</code>, per the <a href="https://edgeguides.rubyonrails.org/active_job_basics.html#create-the-job">ActiveJob docs</a>)</li>
<li>Call the <code class="highlighter-rouge">notify user job</code> from the controller, instead of calling the <code class="highlighter-rouge">user notifier</code> directly.</li>
</ol>
<h3 id="make-the-job">Make the job</h3>
<p>We’ll hand-roll this. Make <code class="highlighter-rouge">app/jobs/send_user_gif_job.rb</code>.</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">SendUserGifJob</span> <span class="o">&lt;</span> <span class="no">ActiveJob</span><span class="o">::</span><span class="no">Base</span>
<span class="n">queue_as</span> <span class="ss">:default</span>
<span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="c1"># do da ting</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>(Deviated slightly from the docs with <code class="highlighter-rouge">ActiveJob::Base</code>. I’m working with Rails 4.2)</p>
<h3 id="make-a-test">Make a test</h3>
<p>Working through the <a href="https://edgeguides.rubyonrails.org/testing.html#testing-jobs">rubyonrails.org docs on testing jobs</a>, I’ll set up the following:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># test/jobs/send_user_gif_job_test.rb</span>
<span class="nb">require</span> <span class="s1">'test_helper'</span>
<span class="k">class</span> <span class="nc">SendUserGifJobTest</span> <span class="o">&lt;</span> <span class="no">ActiveJob</span><span class="o">::</span><span class="no">TestCase</span>
<span class="nb">test</span> <span class="s1">'that email is sent'</span> <span class="k">do</span>
<span class="no">SendUserGifJob</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span><span class="s2">"test@test.com"</span><span class="p">,</span> <span class="s2">"hello"</span><span class="p">)</span>
<span class="c1"># literally no idea what to assert here...</span>
<span class="c1"># assert </span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>I’ve no idea what to assert just yet, but we’ll get there. Lets run the test!</p>
<p>…</p>
<p>Unfortunately, this test passes. :(</p>
<p>After taking a look at the <a href="https://github.com/mperham/sidekiq/wiki/Testing">testing Sidekiq</a> docs, I’ve got some ideas.</p>
<h3 id="messed-up-sidekiq">Messed up Sidekiq?</h3>
<p>After a bit of playing in the <code class="highlighter-rouge">rails console</code>, I had a bunch of bad jobs that Sidekiq was trying to process. Every time I started Sidekiq, it broke with a stack trace for “uninitialized constant”, for a job/class/worker that didn’t exist.</p>
<p>To clear out everything in Sidekiq, run the following from the rails console:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Sidekiq::Queue.all.each(&amp;:clear)
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Sidekiq::DeadSet.new.clear
</code></pre></div></div>
<p>As usual, I found the answer <a href="https://stackoverflow.com/a/47290191/3210178">on Stack Overflow</a> (I could see this being a very dangerous command to run in any sort of production environment. Don’t do that, please.)</p>
<p>After clearing out the queue, I can run Sidekiq just fine.</p>
<h3 id="reworked-the-test-and-worker">Reworked the test and worker</h3>
<p>These are <em>workers</em> and not <em>jobs</em>. <code class="highlighter-rouge">ActiveJob</code> <em>jobs</em> live in <code class="highlighter-rouge">/jobs</code>. So, if you want a <em>worker</em>, don’t put it in the <code class="highlighter-rouge">/jobs</code> directory, put it in the <code class="highlighter-rouge">/workers</code> directory.</p>
<p>Don’t ask me why I know this.</p>
<p>I had problems because of where I stuck these files, and <s>had some other problems because of naming conventions</s> decided naming conventions are important.</p>
<p>So, I threw away all the work and did <code class="highlighter-rouge">rails g sidekiq:worker SendGifToUserWorker</code>.</p>
<p>Here’s what I’ve got right now, after the <code class="highlighter-rouge">rails g</code> and taking some examples from the testing docs:</p>
<p>My <code class="highlighter-rouge">SendGifToUserWorker</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/workers/send_gif_to_user_worker.rb</span>
<span class="k">class</span> <span class="nc">SendGifToUserWorker</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span>
<span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="c1"># Do something</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>My <code class="highlighter-rouge">SendGifToUserWorkerTest</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># test/workers/send_gif_to_user_worker_test.rb</span>
<span class="nb">require</span> <span class="s1">'test_helper'</span>
<span class="k">class</span> <span class="nc">SendGifToUserWorkerTest</span> <span class="o">&lt;</span> <span class="no">ActiveJob</span><span class="o">::</span><span class="no">TestCase</span>
<span class="nb">test</span> <span class="s1">'that email is sent'</span> <span class="k">do</span>
<span class="no">SendGifToUserWorker</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span><span class="s2">"test@test.com"</span><span class="p">,</span> <span class="s2">"hello"</span><span class="p">)</span>
<span class="c1"># literally no idea what to assert here...</span>
<span class="c1"># assert </span>
<span class="k">end</span>
<span class="nb">test</span> <span class="s1">'that job is pushed to queue'</span> <span class="k">do</span>
<span class="n">assert_equal</span> <span class="mi">0</span><span class="p">,</span> <span class="no">SendGifToUserWorker</span><span class="p">.</span><span class="nf">jobs</span><span class="p">.</span><span class="nf">size</span>
<span class="no">SendGifToUserWorker</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span><span class="s2">"test@test.com"</span><span class="p">,</span> <span class="s2">"hello"</span><span class="p">)</span>
<span class="n">assert_equal</span> <span class="mi">1</span><span class="p">,</span> <span class="no">SendGifToUserWorker</span><span class="p">.</span><span class="nf">jobs</span><span class="p">.</span><span class="nf">size</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Unfortunately, the tests pass. This tells me the job is running fine (I guess), but no clue what is happening under the hood.</p>
<p><em>correction: the second test passes every-other-time or so.</em> The <code class="highlighter-rouge">jobs.size</code> queue isn’t always starting at 0, so it fails the first assertion of 0.</p>
<p>A fix was to add the following setup method:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># test/workers/send_gif_to_user_worker_test.rb:5</span>
<span class="k">def</span> <span class="nf">setup</span>
<span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span><span class="p">.</span><span class="nf">clear_all</span>
<span class="k">end</span>
</code></pre></div></div>
<h3 id="making-sidekiq-do-stuff-via-the-rails-console">Making Sidekiq do stuff via the Rails Console</h3>
<p>Since the tests don’t push <em>actual</em> jobs to Sidekiq, I don’t see any indication that anything interesting is happening in Sidekiq web, or Redis, or the Sidekiq terminal window. :(</p>
<p>I updated the mail model in the application to actually use Sidekiq (no failing test quite right now, sorry) and here’s my worker:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/workers/send_gif_to_user_worker.rb</span>
<span class="k">class</span> <span class="nc">SendGifToUserWorker</span>
<span class="kp">include</span> <span class="no">Sidekiq</span><span class="o">::</span><span class="no">Worker</span>
<span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">thought</span><span class="p">)</span>
<span class="no">UserNotifier</span><span class="p">.</span><span class="nf">send_randomness_email</span><span class="p">(</span><span class="n">email</span><span class="p">,</span> <span class="n">thought</span><span class="p">).</span><span class="nf">deliver_now</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And it’s getting called from the mailers controller, like so:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/controllers/mailers_controller.rb</span>
<span class="k">class</span> <span class="nc">MailersController</span> <span class="o">&lt;</span> <span class="no">ApplicationController</span>
<span class="k">def</span> <span class="nf">create</span>
<span class="no">SendGifToUserWorker</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span><span class="n">params</span><span class="p">[</span><span class="ss">:mailers</span><span class="p">][</span><span class="ss">:email</span><span class="p">],</span> <span class="n">params</span><span class="p">[</span><span class="ss">:mailers</span><span class="p">][</span><span class="ss">:thought</span><span class="p">])</span>
<span class="n">flash</span><span class="p">[</span><span class="ss">:message</span><span class="p">]</span> <span class="o">=</span> <span class="s2">"You did it! Email sent to </span><span class="si">#{</span><span class="n">params</span><span class="p">[</span><span class="ss">:mailers</span><span class="p">][</span><span class="ss">:email</span><span class="p">]</span><span class="si">}</span><span class="s2">"</span>
<span class="n">redirect_to</span> <span class="s2">"/sent"</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">sent</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>With that setup, in my <code class="highlighter-rouge">rails console</code>, I can do something like <code class="highlighter-rouge">SendGifToUserWorker.perform_async("test@test.com", "hello")</code>, and I get back some sort of GUID:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>main:0&gt; SendGifToUserWorker.perform_async("test@test.com", "hello")
=&gt; "08e6a309cf7c46dc0178c53f"
main:0&gt; SendGifToUserWorker.perform_async("test@test.com", "hello")
=&gt; "8b962d28217ae177564f0fd7"
</code></pre></div></div>
<p>Each of these talks to Sidekiq, and you can see these jobs go by in the logs:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2018-07-27T17:13:55.023Z 10221 TID-ovusw76r0 SendGifToUserWorker JID-08e6a309cf7c46dc0178c53f INFO: start
2018-07-27T17:13:55.023Z 10221 TID-ovusw76r0 SendGifToUserWorker JID-08e6a309cf7c46dc0178c53f INFO: done: 0.0 sec
2018-07-27T17:13:57.521Z 10221 TID-ovusw781o SendGifToUserWorker JID-8b962d28217ae177564f0fd7 INFO: start
2018-07-27T17:13:57.521Z 10221 TID-ovusw781o SendGifToUserWorker JID-8b962d28217ae177564f0fd7 INFO: done: 0.0 sec
</code></pre></div></div>
<p>This is what it looks like, in the logs and sidekiq web, running the jobs from the Rails console:</p>
<p><img src="/images/2018-07-27_sidekiq_rails_console.gif" alt="rails console, sidekiq web, and sidekiq" /></p>
<p>So, cool. My job still isn’t doing anything, but at least it’s running. I guess.</p>
<h3 id="restart-sidekiq-when-you-make-a-change-to-a-worker">Restart Sidekiq when you make a change to a worker</h3>
<p>It makes sense that the Sidekiq worker test might assert JUST that jobs get queued correctly.</p>
<p>I’m still not content - my tests are passing, <em>without the sidekiq worker actually doing anything</em>. I’d feel great about a red test related to it.</p>
<p>Everything to this point <a href="https://github.com/josh-works/turing_sidekiq_tutorial/tree/38f5750293edf3198e11b114851c8d313608f334">is on commit <code class="highlighter-rouge">38f5750</code></a>, if you’re following along.</p>
<p>Sidekiq is queuing the job as I’d expect it to, even though it’s not doing anything.</p>
<p>.</p>
<p>.</p>
<p>.</p>
<p>I just spent an embarrassing amount of time “troubleshooting” why my worker wasn’t doing what I thought it should do. Turns out <em>you need to restart Sidekiq if you change a Sidekiq job</em>. Maybe this isn’t always true, but if you’re saying</p>
<blockquote>
<p>why isn’t <new thing=""> showing up in Sidekiq?</new></p>
</blockquote>
<p>just restart Sidekiq.</p>
<h2 id="watching-redis">Watching Redis</h2>
<p>I want to make sure that this stuff is getting in and out of Redis as expected. Redis is a super fast key:value store, and we should see stuff getting written to, and read from Redis.</p>
<p>Use <code class="highlighter-rouge">redis-server</code> to start Redis running. It’s pretty boring to watch, and doesn’t show you any information about data placed in/out of it, so not that helpful.</p>
<p><img src="/images/2018-07-28_redis_01.jpg" alt="boring redis" /></p>
<p>Once you have <code class="highlighter-rouge">redis-server</code> running, you can run (in another terminal tab) <code class="highlighter-rouge">redis-cli monitor</code>, which dumps you into something that prints a TON of logs when it’s not doing anything. (all of the <code class="highlighter-rouge">hmset macbookpro</code> stuff is still <em>me doing nothing</em>. It’s reading “server status” details like a hyperactive chipmunk on crack.)</p>
<p><img src="/images/2018-07-28_redis_02.gif" alt="slow your roll, Redis" /></p>
<p>I found the signal-to-noise ratio of <code class="highlighter-rouge">redis-cli monitor</code> to make it near-useless. What we care about in Redis are <code class="highlighter-rouge">hset</code>, and <code class="highlighter-rouge">lpush</code>. Maybe more, but this is sufficient for finding what I want.</p>
<p>So, once you’ve got redis running, to watch the logs for JUST <code class="highlighter-rouge">hset</code>s and <code class="highlighter-rouge">lpush</code>es, run:</p>
<p><code class="highlighter-rouge">redis-cli monitor | grep -E "(hset|lpush)"</code></p>
<p>And you’ll see nothing, until Sidekiq pushes jobs to Redis, and reads from it:</p>
<p><img src="/images/2018-07-28_redis_03.gif" alt="thank you, redis" /></p>
<ul>
<li><a href="https://redis.io/commands/hset">redis docs for <code class="highlighter-rouge">hset</code></a></li>
<li><a href="https://redis.io/commands/lpush">redis docs for <code class="highlighter-rouge">lpush</code></a></li>
</ul>
<p>Here’s those lines, formatted for easier reading:</p>
<h4 id="lpush">lpush</h4>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="mf">1532784329.095661</span> <span class="p">[</span><span class="mi">0</span> <span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="p">:</span><span class="mi">53832</span><span class="p">]</span> <span class="n">lpush</span> <span class="n">queue</span><span class="ss">:default</span>
<span class="p">{</span>
<span class="k">class</span><span class="ss">:SendGifToUserWorker</span><span class="p">,</span>
<span class="n">args</span><span class="p">:[</span><span class="n">flip</span><span class="p">,</span><span class="n">flop</span><span class="p">],</span>
<span class="k">retry</span><span class="ss">:true</span><span class="p">,</span>
<span class="n">queue</span><span class="ss">:default</span><span class="p">,</span>
<span class="n">jid</span><span class="ss">:adfa15f29ed6c09cda7f6973</span><span class="p">,</span>
<span class="n">created_at</span><span class="p">:</span><span class="mf">1532784329.095427</span><span class="p">,</span>
<span class="n">enqueued_at</span><span class="p">:</span><span class="mf">1532784329.095466</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="hset">hset</h4>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="mf">1532784332.778327</span> <span class="p">[</span><span class="mi">0</span> <span class="mf">127.0</span><span class="o">.</span><span class="mf">0.1</span><span class="p">:</span><span class="mi">53803</span><span class="p">]</span> <span class="n">hset</span> <span class="no">MacBook</span><span class="o">-</span><span class="no">Pro</span><span class="o">-</span><span class="mi">6715</span><span class="p">.</span><span class="nf">local</span><span class="p">:</span><span class="mi">32134</span><span class="ss">:cc8d1568c5c6:workers</span> <span class="n">ow3kb5tjc</span>
<span class="p">{</span>
<span class="n">queue</span><span class="ss">:default</span><span class="p">,</span>
<span class="ss">payload:
</span><span class="p">{</span>
<span class="k">class</span><span class="ss">:SendGifToUserWorker</span><span class="p">,</span>
<span class="n">args</span><span class="p">:[</span><span class="n">flip</span><span class="p">,</span><span class="n">flop</span><span class="p">],</span>
<span class="k">retry</span><span class="ss">:true</span><span class="p">,</span>
<span class="n">queue</span><span class="ss">:default</span><span class="p">,</span>
<span class="n">jid</span><span class="ss">:adfa15f29ed6c09cda7f6973</span><span class="p">,</span>
<span class="n">created_at</span><span class="p">:</span><span class="mf">1532784329.095427</span><span class="p">,</span>
<span class="n">enqueued_at</span><span class="p">:</span><span class="mf">1532784329.095466</span>
<span class="p">},</span>
<span class="n">run_at</span><span class="p">:</span><span class="mi">1532784329</span>
<span class="p">}</span>
</code></pre></div></div>
<p>By the way, Redis is a bit cleaner if you run the server as a background process. To do this, do:</p>
<p><code class="highlighter-rouge">redis-server &amp;</code> (the <code class="highlighter-rouge">&amp;</code> makes it a background process).</p>
<p>To stop Redis, just do <code class="highlighter-rouge">redis-cli shutdown</code></p>
<h3 id="what-doesdoes-not-occur">What does/does not occur</h3>
<p>So, how can we be sure that our Sidekiq job is actually firing? Lets see what it looks like, using this worker <em>with</em> Sidekiq, and without.</p>
<p>Compare these two lines:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">SendGifToUserWorker</span><span class="p">.</span><span class="nf">new</span><span class="p">.</span><span class="nf">perform</span><span class="p">(</span><span class="n">params</span><span class="p">[</span><span class="ss">:mailers</span><span class="p">][</span><span class="ss">:email</span><span class="p">],</span> <span class="n">params</span><span class="p">[</span><span class="ss">:mailers</span><span class="p">][</span><span class="ss">:thought</span><span class="p">])</span>
<span class="no">SendGifToUserWorker</span><span class="p">.</span><span class="nf">perform_async</span><span class="p">(</span><span class="n">params</span><span class="p">[</span><span class="ss">:mailers</span><span class="p">][</span><span class="ss">:email</span><span class="p">],</span> <span class="n">params</span><span class="p">[</span><span class="ss">:mailers</span><span class="p">][</span><span class="ss">:thought</span><span class="p">])</span>
</code></pre></div></div>
<p>Which one is using Sidekiq?</p>
<p>The first line is creating a new instance of the worker class, and calling <code class="highlighter-rouge">perform</code> directly on it. <em>It’s not hitting Sidekiq</em>, and if the job fails, it won’t requeue.</p>
<p>At least, I think this is the case.</p>
<p>Here’s what <code class="highlighter-rouge">rails server</code> (left), redis (filtering for <code class="highlighter-rouge">hset</code> and <code class="highlighter-rouge">lpush</code> events, top right) and Sidekiq (bottom right) look like when I trigger the email worker:</p>
<p><img src="/images/2018-07-30_sidekiq_02.gif" alt="using sidekiq" title="using sidekiq" /></p>
<p>Here’s how I can see a job using <code class="highlighter-rouge">Worker.new.perform</code> doesn’t “hit” sidekiq:</p>
<p><img src="/images/2018-07-30_sidekiq_01.gif" alt="not using sidekiq" title="not using sidekiq" /></p>
<p>The Rails server just sits for a while, and redis and Sidekiq seem clueless of the event?</p>
<h3 id="conclusion">Conclusion</h3>
<p>One of the bugs I was working with, the root of the problem was the worker/job seemed to not be requeuing when it failed. I spent time touching up the worker, and logging around the specific error (it was a timeout on an internal API endpoint), but even after rescuing the timeout, the job wasn’t happening again.</p>
<p>Someone else on the team wisely pointed out that the entire worker was getting called with <code class="highlighter-rouge">Worker.new.perform()</code>, which was sidestepping (oh, the puns) the Sidekiq infrastructure. I understood this to be true, but I still wanted to see it for myself.</p>
<p>In the process of working through this lesson, I found a <a href="https://stackoverflow.com/questions/19251976/get-sidekiq-to-execute-a-job-immediately">Stack Overflow post</a> about “how to make Sidekiq execute a job immediately”. The only answer suggested calling <code class="highlighter-rouge">Worker.new.perform</code>, and that must be, for some reason, why this bit of code ended up in our codebase. The developer wanted it to happen immediately, and perhaps did not expect it to fail at any point.</p>
<p>Anyway, I’m content that I understand the basic difference between <code class="highlighter-rouge">Worker.perform_async</code> and <code class="highlighter-rouge">Worker.new.perform</code>, and next time I encounter an issue like this, I’ll much more quicly wrap my head around it.</p>
<h3 id="resources">Resources</h3>
<ul>
<li><a href="http://backend.turing.io/module3/lessons/intro_to_background_workers">Turing Sidekiq lesson</a></li>
<li><a href="https://www.engineyard.com/blog/testing-async-emails-rails-42">Testing async emails, the Rails 4.2+ way</a></li>
<li><a href="https://stackoverflow.com/questions/30119144/rails-how-to-switch-between-dev-and-production-mode">Quickly booting <code class="highlighter-rouge">rails s</code> into production</a></li>
<li><a href="https://stackoverflow.com/questions/7829480/no-route-matches-get-assets">No route matches [GET] /assets</a></li>
<li><a href="https://stackoverflow.com/questions/14713084/how-to-see-set-get-in-redis-log">StackOverflow: How to see set/get/ in redis log</a></li>
</ul>Josh Thompsonthompsonjoshd@gmail.comI’ve recently had to learn more about background jobs (using Sidekiq, specifically) for some bugs I was working on. I learned a lot. Much of it was extremely basic. Anyone who knows much at all about Sidekiq will say “oh, duh, of course that’s true”, but at the time, it wasn’t obvious to me. The reason I needed such basic overviews is because prior to my current job, I’d had just a few hours of exposure to background jobs, and understood little of those hours. And I got dropped into a project that has dozens of jobs, handling hundreds of thousands of actions a day. As is my style, when I don’t understand something, I like to go to the very basics. Most of the interesting stuff is way down at the bottom, on watching Redis do it’s thingMocks &amp; Stubs &amp; Exceptions in Ruby2018-05-26T13:00:00+00:002018-05-26T13:00:00+00:00https://josh.works/mocks_stubs_exceptions_ruby<p>Some of my recent work has been around improving error handling and logging.</p>
<p>We had some tasks that, if they failed to execute correctly, were <em>supposed</em> to raise exceptions, log themselves, and re-queue, but they were not.</p>
<p>The class in which I was working managed in large part API calls to external services, services that our team has no control over. Sometimes the services work great, sometimes they don’t. The tests for this class made heavy use of mocks, stubs, and the <a href="https://github.com/vcr/vcr">VCR gem</a>.</p>
<p>This post was originally going to be about error handling, but then I realized I was getting a bit crossed up just by the mocks and stubs, so I took a quick detour into the topic, in order to build out a mental model of what was going on.</p>
<p>As I often do, I spun up a bare-bones implementation of all of the required pieces, to see how everything was playing together. <a href="https://semaphoreci.com/community/tutorials/mocking-in-ruby-with-minitest">One guide</a> I found was excellent, but the sample app was a bit over-kill. I didn’t want a whole new rails app - I just wanted a class and test file.</p>
<p><em>note: This isn’t a tutorial, per se, but it’s just two files and you can easily copy-paste the code into an editor and run the tests. I’ll link to specific commits in a github repo throughout. Clone it down, check out the commit, poke around.</em></p>
<p>This project took me through:</p>
<ul>
<li>stubbing</li>
<li>mocking</li>
<li>raising exceptions</li>
<li>rescuing exceptions</li>
<li>testing all of the above
<!--more--></li>
</ul>
<h1 id="what-are-mocks-and-stubs">What are mocks and stubs?</h1>
<p>I’ll admit - the “standard definition” of mocks and stubs didn’t mean much to me. Quite a few definitions I found referenced “<a href="https://martinfowler.com/articles/mocksArentStubs.html">the Fowler article</a>” which, while interesting, doesn’t quite move me forward on testing Rails app <em>today</em>.</p>
<p>From <a href="https://stackoverflow.com/a/5164709/3210178">StackOverflow</a></p>
<blockquote>
<ul>
<li>mocks are objects that have a similar interface as something else</li>
<li>stubs are fake methods and return a specific answer</li>
</ul>
</blockquote>
<p>So, of course, I needed to see them function ‘in the wild’.</p>
<h1 id="eliminate-dependencies-on-other-classes-in-testing">Eliminate dependencies on other classes in testing</h1>
<p>I decided to make up this <code class="highlighter-rouge">service</code> and <code class="highlighter-rouge">connection</code> class.</p>
<p>Imagine I want to connect to a few different third-party services. I’ll have a <code class="highlighter-rouge">connection</code> class into which I can pass a third-party service “object”, and it should “connect”.</p>
<p>I will assume I can call <code class="highlighter-rouge">third_party_service.status</code> and get back <code class="highlighter-rouge">200</code>, <code class="highlighter-rouge">404</code>, <code class="highlighter-rouge">504</code>, etc.</p>
<p>I will also assume that sometimes <em>my own service</em> will not function correctly, and so <code class="highlighter-rouge">my_service.status</code> can also be any particular status code, like <code class="highlighter-rouge">200</code>, <code class="highlighter-rouge">404</code>, <code class="highlighter-rouge">504</code>, etc.</p>
<p>I’d like to be able to test that I can raise specific errors if either <code class="highlighter-rouge">connection.status</code> or <code class="highlighter-rouge">third_party_service.status</code> is not <code class="highlighter-rouge">2xx</code>.</p>
<p>Here’s my sample <code class="highlighter-rouge">Connection</code> class, where I’ve hard-coded the 200 status:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Connection</span>
<span class="nb">attr_reader</span> <span class="ss">:status_code</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@status_code</span> <span class="o">=</span> <span class="n">get_service_status</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">get_service_status</span>
<span class="mi">200</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">srvc</span><span class="p">)</span>
<span class="k">return</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And here’s the test that shows all is groovy:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ConnectionTest</span> <span class="o">&lt;</span> <span class="no">Minitest</span><span class="o">::</span><span class="no">Test</span>
<span class="k">def</span> <span class="nf">test_initializes_with_response_code</span>
<span class="n">c</span> <span class="o">=</span> <span class="no">Connection</span><span class="p">.</span><span class="nf">new</span>
<span class="n">assert_equal</span> <span class="mi">200</span><span class="p">,</span> <span class="n">c</span><span class="p">.</span><span class="nf">status_code</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Now, lets say I’ve got this third party service. Since I’m testing this all locally, and I want to square away mocking and stubbing anyway, I’m getting real fancy and I’m putting this in the same file as my <code class="highlighter-rouge">Connection</code> class:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Connection</span>
<span class="nb">attr_reader</span> <span class="ss">:status_code</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@status_code</span> <span class="o">=</span> <span class="n">get_service_status</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">get_service_status</span>
<span class="mi">200</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">srvc</span><span class="p">)</span>
<span class="k">return</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># this is my third-party class that `Connection` might have dependencies upon</span>
<span class="k">class</span> <span class="nc">Service</span>
<span class="nb">attr_reader</span> <span class="ss">:status_code</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@status_code</span> <span class="o">=</span> <span class="n">get_service_status</span>
<span class="c1"># status_code would be 200, 504, 404, etc</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">get_service_status</span>
<span class="c1"># do complicated stuff here to get status code</span>
<span class="c1"># return a status code</span>
<span class="mi">200</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Inside my test, when I call <code class="highlighter-rouge">Connection.new.connect_to_external_service()</code>, I need to pass in a service object, like so:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>srvc = Service.new
conn = Connection.new
conn.connect_to_external_service(srvc)
</code></pre></div></div>
<p>And I’m off to the races. This is where I want to start raising exceptions if either class returns errors, and where stubbing and mocking comes to be very valuable.</p>
<p>So, this test would pass:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">ConnectionTest</span> <span class="o">&lt;</span> <span class="no">Minitest</span><span class="o">::</span><span class="no">Test</span>
<span class="k">def</span> <span class="nf">test_connection_default_status_code_is_200</span>
<span class="n">c</span> <span class="o">=</span> <span class="no">Connection</span><span class="p">.</span><span class="nf">new</span>
<span class="n">assert_equal</span> <span class="mi">200</span><span class="p">,</span> <span class="n">c</span><span class="p">.</span><span class="nf">status_code</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">test_external_service_default_status_code_is_200</span>
<span class="n">conn</span> <span class="o">=</span> <span class="no">Connection</span><span class="p">.</span><span class="nf">new</span>
<span class="n">srvc</span> <span class="o">=</span> <span class="no">Service</span><span class="p">.</span><span class="nf">new</span>
<span class="n">assert_equal</span> <span class="mi">200</span><span class="p">,</span> <span class="n">conn</span><span class="p">.</span><span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">srvc</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>But I want to make <code class="highlighter-rouge">srvc.status_code</code> be something besides <code class="highlighter-rouge">200</code>.</p>
<p>Lets figure out how to make <code class="highlighter-rouge">srvc.status_code</code> be 404, and maybe in the process remove the dependency upon the <code class="highlighter-rouge">Service</code> class all together.</p>
<h1 id="stubbing-the-service-object-to-force-errors">Stubbing the service object to force errors</h1>
<p><a href="https://github.com/josh-works/exception_practice/tree/dd54c0f14e3b8c5c8dff9bd1666fea8686ed718c">Current github commit</a></p>
<p>Right now, I want to run a test like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_raises_error_when_service_returns_4xx_no_mocks</span>
<span class="n">srvc</span> <span class="o">=</span> <span class="no">Service</span><span class="p">.</span><span class="nf">new</span>
<span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span> <span class="o">=</span> <span class="mi">404</span>
<span class="n">conn</span> <span class="o">=</span> <span class="no">Connection</span><span class="p">.</span><span class="nf">new</span>
<span class="n">assert_raises</span> <span class="no">Connection</span><span class="o">::</span><span class="no">ServiceNotFound</span> <span class="k">do</span>
<span class="n">conn</span><span class="p">.</span><span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">srvc</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This won’t work because I don’t have an <code class="highlighter-rouge">attr_writer</code> or <code class="highlighter-rouge">attr_accessor</code> for the <code class="highlighter-rouge">Service</code> instance variable of <code class="highlighter-rouge">status_code</code></p>
<p>This is the error I get when I run the test:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1) Error:
ConnectionTest#test_raises_error_when_service_returns_4xx_no_mocks:
NoMethodError: undefined method `status_code=' for #&lt;Service:0x00007f81ce3c34d8 @status_code=200&gt;
</code></pre></div></div>
<p>And this makes sense, right? <em>I cannot assume ownership of the third party service</em>.</p>
<p>Here’s how I can stub out the method as I want it, using <code class="highlighter-rouge">mocha</code>’s tooling for stubbing:</p>
<p><code class="highlighter-rouge">gem install mocha</code>, and include the following front-matter in the test file:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'./lib/connection'</span>
<span class="nb">require</span> <span class="s1">'minitest/autorun'</span>
<span class="nb">require</span> <span class="s1">'mocha/minitest'</span>
<span class="k">class</span> <span class="nc">ConnectionTest</span> <span class="o">&lt;</span> <span class="no">Minitest</span><span class="o">::</span><span class="no">Test</span>
<span class="k">def</span> <span class="nf">test_raise_error_when_service_returns_4xx_using_stubs</span>
<span class="n">serv</span> <span class="o">=</span> <span class="no">Service</span><span class="p">.</span><span class="nf">new</span>
<span class="c1"># instantiating a new Service object. if I call</span>
<span class="c1"># serv.status_code, it would return 200</span>
<span class="no">Service</span><span class="p">.</span><span class="nf">any_instance</span><span class="p">.</span><span class="nf">stubs</span><span class="p">(</span><span class="ss">:status_code</span><span class="p">).</span><span class="nf">returns</span><span class="p">(</span><span class="mi">404</span><span class="p">)</span>
<span class="c1"># override the method to always return 404</span>
<span class="n">conn</span> <span class="o">=</span> <span class="no">Connection</span><span class="p">.</span><span class="nf">new</span>
<span class="n">assert_raises</span> <span class="no">Connection</span><span class="o">::</span><span class="no">ServiceNotFound</span> <span class="k">do</span>
<span class="n">conn</span><span class="p">.</span><span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">serv</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The magic is <code class="highlighter-rouge">Service.any_instance.stubs(:status_code).returns(404)</code></p>
<p><em>Stubbing modifies an existing object to coerce it into giving a certain output.</em></p>
<p>This is cool, but nothing revelatory to me. I’ve been working with stubs for a while. What finally clicked for me was the difference between <em>mocks</em> and stubs.</p>
<p>The above test still requires access to a <code class="highlighter-rouge">Service</code> class. If Service doesn’t exist, or requires any setup that I didn’t pass it, I’m out of luck and will need to do <em>even more</em> test setup.</p>
<p>Enter mocks.</p>
<h1 id="mocking-the-service-object">Mocking the service object</h1>
<p>I want to remove <em>all</em> dependencies on <code class="highlighter-rouge">Service</code>. I want to create a service object in my test, assign it variables, and use them in my <code class="highlighter-rouge">Connection</code> class, all without <code class="highlighter-rouge">Connection</code> knowing anything about <code class="highlighter-rouge">Service</code>, or indeed, <code class="highlighter-rouge">Service</code> knowing anything about itself.</p>
<p>We’ve got <code class="highlighter-rouge">Connection</code> under test here, remember. Not <code class="highlighter-rouge">Service</code></p>
<p>Here’s the commit for the following code: <a href="https://github.com/josh-works/exception_practice/tree/6b7014cb0c2bdfefe5a7b0eae633e2f2499d4ffb">6b7014c</a></p>
<p>I’ll make <code class="highlighter-rouge">srvc</code> a <code class="highlighter-rouge">stub</code>, which I can now make it do <em>anything I want</em>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_rais_error_when_service_returns_4xx_using_mocks</span>
<span class="n">conn</span> <span class="o">=</span> <span class="no">Connection</span><span class="p">.</span><span class="nf">new</span>
<span class="n">srvc</span> <span class="o">=</span> <span class="n">stub</span><span class="p">(</span><span class="s2">"srvc is now a stub."</span><span class="p">)</span>
<span class="c1"># srvc.class == Mocha::Mock</span>
<span class="n">srvc</span><span class="p">.</span><span class="nf">stubs</span><span class="p">(</span><span class="ss">:status_code</span><span class="p">).</span><span class="nf">returns</span><span class="p">(</span><span class="mi">404</span><span class="p">)</span>
<span class="n">assert_raises</span> <span class="no">Connection</span><span class="o">::</span><span class="no">ServiceNotFound</span> <span class="k">do</span>
<span class="n">conn</span><span class="p">.</span><span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">srvc</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>so I no longer have a <code class="highlighter-rouge">Service</code> object. If I call <code class="highlighter-rouge">srvc.class</code> on that stubbed object, it’s <code class="highlighter-rouge">Mocha::Mock</code></p>
<p>I then assign it a method of <code class="highlighter-rouge">.status_code</code>, and tell it to return <code class="highlighter-rouge">404</code>. Boom. Done.</p>
<p>So, I’m no longer dependent upon the <code class="highlighter-rouge">Service</code> model for any of my <code class="highlighter-rouge">Connection</code> testing. I’d say thats a win.</p>
<p><a href="https://github.com/josh-works/exception_practice/blob/98850fbf764c974724fea0f280379cc07918eba9/test/connection_test.rb">Here’s the commit for making full use of mocks</a></p>
<h4 id="youll-never-not-necessarily-see-the-word-mock-in-your-code-even-when-using-mocks">You’ll <strike>never</strike> not necessarily see the word ‘mock’ in your code, even when using mocks.</h4>
<p>I’m not the sharpest tool in the shed, so I do a lot of pattern matching. Articles about stubbing always had the word ‘stub’ being scattered about the code. That made it easy to see when a stub was being used.</p>
<p>Mocks, though, don’t necessarily get called in the code. Traditionally, you could do <code class="highlighter-rouge">obj = stub('object')</code>, and get your stubbed object that way. Now (I think this is more recent) you can make it a bit more explicit and call <code class="highlighter-rouge">obj = mock('object')</code>, and you’ll be on your way.</p>
<p>Either way, <code class="highlighter-rouge">mock()</code> and <code class="highlighter-rouge">stub()</code> are interchangeable:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mocked_object</span> <span class="o">=</span> <span class="n">mock</span><span class="p">(</span><span class="s1">'mocked_object'</span><span class="p">)</span>
<span class="n">stubbed_object</span> <span class="o">=</span> <span class="n">stub</span><span class="p">(</span><span class="s1">'stubbed_object'</span><span class="p">)</span>
</code></pre></div></div>
<p>gives:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&gt; mocked_object
=&gt; #&lt;Mock:mocked_object&gt;
&gt; stubbed_object
=&gt; #&lt;Mock:stubbed_object&gt;
</code></pre></div></div>
<p>So, if you see <code class="highlighter-rouge">foo = mock()</code>, it’s a mocked object, but if you see <code class="highlighter-rouge">foo = stub()</code>, it’s <em>also</em> a mocked object.</p>
<h1 id="exceptions">Exceptions</h1>
<p>We’ve covered mocking and stubbing, and are ready to dig into exception raising and handling.</p>
<p>Lets say my third party service goes down (404) or times out (504). I’d like to raise these descriptive errors, and do something with it.</p>
<p>This is the test I could create:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">test_raise_service_timeout_if_service_returns_504</span>
<span class="vi">@srv</span><span class="p">.</span><span class="nf">stubs</span><span class="p">(</span><span class="ss">:status_code</span><span class="p">).</span><span class="nf">returns</span><span class="p">(</span><span class="mi">504</span><span class="p">)</span>
<span class="n">assert_raises</span> <span class="no">Connection</span><span class="o">::</span><span class="no">ServiceTimeOut</span> <span class="k">do</span>
<span class="vi">@conn</span><span class="p">.</span><span class="nf">connect_to_external_service</span><span class="p">(</span><span class="vi">@srv</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And I can update my <code class="highlighter-rouge">Connection</code> class:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="k">def</span> <span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">srvc</span><span class="p">)</span>
<span class="k">raise</span> <span class="no">ServiceNotFound</span> <span class="k">if</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span> <span class="o">==</span> <span class="mi">404</span>
<span class="k">raise</span> <span class="no">ServiceTimeOut</span> <span class="k">if</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span> <span class="o">==</span> <span class="mi">504</span>
<span class="c1"># do other stuff</span>
<span class="k">return</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This won’t quite pass - for reasons I’m still exploring, we cannot raise a <code class="highlighter-rouge">ServiceTimeOut</code> or <code class="highlighter-rouge">ServiceNotFound</code> error unless these classes are included in the class:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">.</span>
<span class="nf">.</span>
<span class="k">def</span> <span class="nf">connect_to_external_service</span><span class="p">(</span><span class="n">srvc</span><span class="p">)</span>
<span class="k">raise</span> <span class="no">ServiceNotFound</span> <span class="k">if</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span> <span class="o">==</span> <span class="mi">404</span>
<span class="k">raise</span> <span class="no">ServiceTimeOut</span> <span class="k">if</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span> <span class="o">==</span> <span class="mi">504</span>
<span class="c1"># do other stuff</span>
<span class="k">return</span> <span class="n">srvc</span><span class="p">.</span><span class="nf">status_code</span>
<span class="k">end</span>
<span class="k">class</span> <span class="nc">ServiceNotFound</span> <span class="o">&lt;</span> <span class="no">StandardError</span><span class="p">;</span> <span class="k">end</span>
<span class="k">class</span> <span class="nc">ServiceTimeOut</span> <span class="o">&lt;</span> <span class="no">StandardError</span><span class="p">;</span> <span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>And these tests pass.</p>
<p>Why do we need <code class="highlighter-rouge">ServiceNotFound</code> and <code class="highlighter-rouge">ServiceTimeOut</code> to inherit from <code class="highlighter-rouge">StandardError</code>, <em>and</em> be included in the class?</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ServiceNotFound.ancestors
=&gt; [Connection::ServiceNotFound,
StandardError,
Exception,
Object,
Kernel,
BasicObject]
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>StandardError.ancestors
=&gt; [StandardError,
Exception,
Object,
Kernel,
BasicObject]
</code></pre></div></div>
<p><code class="highlighter-rouge">raise</code> is a Kernal method, <a href="http://ruby-doc.org/core-2.5.1/Kernel.html#method-i-raise">it seems</a>.</p>
<p>I’m going to leave this here, for now. The whole problem that led to this blog post I am now fully ready to dig into - I wanted to <code class="highlighter-rouge">rescue</code> some raised exceptions, and when I did that, the standard <code class="highlighter-rouge">assert_raises</code> testing stopped working, because… it seemed the raised error was caught and “squelched” by the rescue, and never bubbled back up to the test.</p>
<p>I’ll write more on that soon.</p>
<h3 id="misc-resources">Misc Resources</h3>
<ul>
<li><a href="https://www.ibm.com/developerworks/library/wa-mockrails/">Mocking and stubbing in Ruby on Rails</a>(this article is from 2007 and is the 2nd ranked result for googling <code class="highlighter-rouge">mocks stubs rails</code>!!!)</li>
<li><a href="https://www.justinweiss.com/articles/testing-network-services-in-ruby/">Testing Network Services in Ruby Is Easier Than You Think</a></li>
<li><a href="https://semaphoreci.com/community/tutorials/mocking-in-ruby-with-minitest">Mocking in Ruby with Minitest</a></li>
</ul>Josh Thompsonthompsonjoshd@gmail.comSome of my recent work has been around improving error handling and logging. We had some tasks that, if they failed to execute correctly, were supposed to raise exceptions, log themselves, and re-queue, but they were not. The class in which I was working managed in large part API calls to external services, services that our team has no control over. Sometimes the services work great, sometimes they don’t. The tests for this class made heavy use of mocks, stubs, and the VCR gem. This post was originally going to be about error handling, but then I realized I was getting a bit crossed up just by the mocks and stubs, so I took a quick detour into the topic, in order to build out a mental model of what was going on. As I often do, I spun up a bare-bones implementation of all of the required pieces, to see how everything was playing together. One guide I found was excellent, but the sample app was a bit over-kill. I didn’t want a whole new rails app - I just wanted a class and test file. note: This isn’t a tutorial, per se, but it’s just two files and you can easily copy-paste the code into an editor and run the tests. I’ll link to specific commits in a github repo throughout. Clone it down, check out the commit, poke around. This project took me through: stubbing mocking raising exceptions rescuing exceptions testing all of the aboveRecommended Reading2018-05-17T13:00:00+00:002018-05-17T13:00:00+00:00https://josh.works/recommended_reading<p>I like to read, and I often recommend books to others.</p>
<p>The recommendations below are entirely non-fiction, but have provided great value to me in very specific ways.</p>
<p>I’ve got ~30 book recommendations grouped into following categories:</p>
<ul>
<li><a href="#thinking-well">Thinking Well</a></li>
<li><a href="#politics">Politics</a></li>
<li><a href="#buildings-and-urbanism">Buildings and Urbanism</a></li>
<li><a href="#theology">Theology</a></li>
<li><a href="#parenting">Parenting</a></li>
<li><a href="#relationships-and-emotions">Relationships</a></li>
<li><a href="#money">Money</a></li>
<li><a href="#uncomfortable-books">Uncomfortable books</a></li>
</ul>
<p>I mention some books “pair well” with other books, just as certain wines pair well with a certain foods. The ideas contained within certain books may compliment (or contrast) the ideas listed in the “pairs with” book.
<!--more--></p>
<h1 id="thinking-well">Thinking Well</h1>
<p>As humans, we do a lot of thinking <sup>[citation needed]</sup>. And, when we’re not thinking, we’re living off of habit, and should think a bit about our habits. We underestimate large risks (driving) and over-estimate small risks (terrorism, kidnapping).</p>
<h3 id="antifragile-things-that-gain-from-disorder"><a href="https://www.goodreads.com/book/show/17340611-antifragile">Antifragile: Things That Gain From Disorder</a></h3>
<p>Most biological systems benefit from stress in the environment. (Muscles. Exercise them, they get stronger). Many man-built institutions generally hate risk and do all they can to avoid exposure to risk. They are successful, for a time, until they catastrophically fail. You can build your life on the principles of a biological system, or on the principles of a rigid institution.</p>
<p><em>Antifragile</em> is an annoying book. The author is a pompous, arrogant man. Unfortunately, he’s also <strong>correct</strong>.</p>
<p>This book, coupled with <a href="#seeing-like-a-state-how-certain-schemes-to-improve-the-human-condition-have-failed">Seeing like a State</a> and <a href="#money-bank-credit-and-economic-cycles">Money, Bank Credit, and Economic Cycles</a> has completely destroyed any confidence I’ve had in the professional thinking class. (Academics, politicians, and others with no “skin in the game”). If you would like to maintain your confidence in these people and institutions, please don’t read these books.</p>
<h3 id="the-black-swan-the-impact-of-the-highly-improbable"><a href="https://www.goodreads.com/book/show/242472.The_Black_Swan">The Black Swan: The Impact of the Highly Improbable</a></h3>
<p>This book is written by the same author of <em>Antifragile</em>, and he talks about life in “mediocrastan” vs “extremistan”, and all the implications of misjudging which world you live in. This book and <em>Antifragile</em> have absolutely shaped many ways in which I live my life.</p>
<p>Taleb is annoying. So many reviews spend the entire review discussing how annoying he is. That’s irrelevant to me - if I can glean substantive value from the ideas, I can put up with an annoying person.</p>
<p>From <a href="https://www.goodreads.com/review/show/27674740?book_show_action=true&amp;from_review_page=1">GoodReads</a>:</p>
<blockquote>
<p><em>The Black Swan</em> deals with the fascinating topic of the nature uncertainty and approaches it from a variety of intellectual angles, mainly the psychological blocks that we are both born with and have created for ourselves that prevent our understanding of the improbable:</p>
<ul>
<li>the narrative fallacy and the problem of induction (the tenuous relationship of cause and effect)</li>
<li>our reliance on flawed mathematical models</li>
<li>the expert problem.</li>
</ul>
<p>Each one of these discussions reinforces his main argument but captivate independently as they are insights to the way we process information.</p>
</blockquote>
<h3 id="a-mind-for-numbers-how-to-excel-at-math-and-science-even-if-you-flunked-algebra"><a href="https://www.goodreads.com/book/show/18693655-a-mind-for-numbers">A Mind for Numbers: How to Excel at Math and Science (Even If You Flunked Algebra)</a></h3>
<p>The “Right brain vs left brain” model is wrong. If you want to learn a technical topic, you can. Just like how if you wanted to cut a piece of wood, you can. If you have the right tool. A butter knife will not cut a piece of wood very well, but a buzz saw might.</p>
<p><em>A Mind for Numbers</em> gives you that buzz saw. With the right tactics and motives, anyone can read anything. I read this before jumping into software development full-time, and this book delivered great value in that endeavor.</p>
<p>From <a href="https://www.goodreads.com/review/show/1080072109?book_show_action=true&amp;from_review_page=1">GoodReads</a>:</p>
<blockquote>
<p>The title of the book doesn’t do it justice. This is a book about how to get good at anything, not just math and science. It’s a light read because it’s full of simple advice. But the stuff it teaches is effective, and I wish it had been taught to me back in 1997 when I was starting graduate school.</p>
<p>If you find yourself checking your phone or screwing around on Facebook while you should be working, read this book. If you’re having trouble learning stuff you need for work at a higher rate than you’re forgetting it, read this book. Do so especially if you’re young, because the longer the time you have left to reap the benefits, the more reading this book is worth to you.</p>
</blockquote>
<h3 id="deep-work-rules-for-focused-success-in-a-distracted-world"><a href="https://www.goodreads.com/book/show/25744928-deep-work">Deep Work: Rules for Focused Success in a Distracted World</a></h3>
<p><em>Deep Work</em> is profound. It was <a href="/i-quit">the best book I read in 2016</a>, and is no small part of the reason I went to Turing and got into software development.</p>
<p>From the author:</p>
<blockquote>
<p>Deep work is the ability to focus without distraction on a cognitively demanding task. It’s a skill that allows you to quickly master complicated information and produce better results in less time. Deep work will make you better at what you do and provide the sense of true fulfillment that comes from craftsmanship. In short, deep work is like a super power in our increasingly competitive twenty-first century economy. And yet, most people have lost the ability to go deep—spending their days instead in a frantic blur of e-mail and social media, not even realizing there’s a better way.</p>
</blockquote>
<p>Pursuing the skill of “deep work” will have a compounding effect on the rest of your life. A little investment now makes the rest of your efforts just a little more effective, and leaves you with more room to get better at deep work, which in turn makes your efforts more effective, and on and on.</p>
<h1 id="politics">Politics</h1>
<p>I’m a fan of the idea of shrinking the political sphere. Politics is where everyone has a rightful sense of ownership of the issue (we all pay taxes, after all) and where the outcomes can be influenced <em>literally by who yells the loudest</em>. It’s embarrassing that the way we organize ourselves relies entirely upon coercion, but for now, that’s what it is. I look forward to when our dominant political institutions go the way of the church at the end of the medieval ages.</p>
<p>A theme through the next few book recommendations is <a href="https://www.fs.blog/2018/02/unintended-consequences/">unintended consequences</a> and <a href="https://www.fs.blog/2016/04/second-level-thinking/">second order effects</a>.</p>
<h3 id="seeing-like-a-state-how-certain-schemes-to-improve-the-human-condition-have-failed"><a href="https://www.goodreads.com/book/show/20186.Seeing_Like_a_State">Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed</a></h3>
<p>The craft of statehood is hard, and the state has been remarkably consistent at sustaining itself, despite myriad harms heaped upon its citizens.</p>
<p>I’m quoting at length from <a href="http://slatestarcodex.com/2017/03/16/book-review-seeing-like-a-state/">Slate Star Codex’s excellent review of the book</a>:</p>
<blockquote>
<p>[The author] starts with the story of “scientific forestry” in 18th century Prussia. Enlightenment rationalists noticed that peasants were just cutting down whatever trees happened to grow in the forests, like a chump. They came up with a better idea: clear all the forests and replace them by planting identical copies of Norway spruce (the highest-lumber-yield-per-unit-time tree) in an evenly-spaced rectangular grid. Then you could just walk in with an axe one day and chop down like a zillion trees an hour and have more timber than you could possibly ever want.</p>
<p>This went poorly. The impoverished ecosystem couldn’t support the game animals and medicinal herbs that sustained the surrounding peasant villages, and they suffered an economic collapse. The endless rows of identical trees were a perfect breeding ground for plant diseases and forest fires. And the complex ecological processes that sustained the soil stopped working, so after a generation the Norway spruces grew stunted and malnourished. Yet for some reason, everyone involved got promoted, and “scientific forestry” spread across Europe and the world.</p>
<p>And this pattern repeats with suspicious regularity across history, not just in biological systems but also in social ones.</p>
</blockquote>
<h3 id="evicted-poverty-and-profit-in-the-american-city"><a href="https://www.goodreads.com/book/show/25852784-evicted">Evicted: Poverty and Profit in the American City</a></h3>
<p>This is a brutal read. It’s not <em>technically</em> illegal to be poor in the USA, but from the perspective of many, it may as well be.</p>
<p>From <a href="https://www.goodreads.com/review/show/1596100376">GoodReads</a>:</p>
<blockquote>
<p>I didn’t realize until I read the afterward that the author of this book put himself right into the middle of the people he portrays lives. He gave them rides to look for houses, he even loaned them small amounts of money at times. He admits that he misses living in the trailer park among them.</p>
<p>This book. I hope more people get it and read it. I’ve been on a “smart book” kick lately and I’ve starred them all pretty highly but this one is just amazing. Desmond knocks it out of the ballpark. You can tell he puts his whole heart into telling these stories.</p>
<p>Now the stories..they are real people: You have to keep reminding yourself of that as you read this book because no one is perfect, they all mess up and the writing is so good that you feel like you are just reading a really good work of fiction.</p>
</blockquote>
<p>The whole review is worth reading.</p>
<h3 id="the-color-of-law"><a href="https://www.goodreads.com/book/show/32191706-the-color-of-law">The Color of Law</a></h3>
<p>The modern US implementation of zoning is a by-product of the Supreme Court making <a href="https://en.wikipedia.org/wiki/Redlining">redlining</a> illegal. Since explicit racism was no longer legal, racist people implemented laws that were not explicitly racist, but everyone understood it to be laws to keep black (and and poor) people out of their neighborhoods.</p>
<p>I’ll state that again - your neighborhoods HOA, your single-family-home neighborhood, your mortgage interest tax-deduction - all this was put in place by racists, to further their vision of a segregated society.</p>
<p>Of course, these policies are harmful to minorities and poor people, <em>but they are also harmful to everyone else, including those who ostensibly benefit from the policies</em>. The US is collectively shooting itself in the foot, every day, by allowing these policies to stand.</p>
<p>From <a href="https://www.goodreads.com/review/show/1994904156?book_show_action=true&amp;from_review_page=1">GoodReads</a>:</p>
<blockquote>
<p>[The Color of Law is the history] of the development of <a href="https://en.wikipedia.org/wiki/De_jure">de jure</a> segregation in the United States - that is, the deliberate result of federal laws and local policies.</p>
<p>[The author] demonstrates convincingly that the problem is entrenched within multiple organizations and legal standards.</p>
<p>For example, the Federal Housing Administration, part of the New Deal set of domestic programs, <em>required segregation in order to qualify for low-interest financing</em>. (emphasis added)</p>
<p>The government-sponsored HOLC required private real-estate agents to appraise neighborhoods and lower pricing values based on the racial composition of its inhabitants.</p>
<p>This of course extends beyond the federal level. State governments, city councils, neighborhoods’ associations, cooperation with non-profits or NGOs that promoted segregation, banks refusing to provide funding to black applicants, landlords charging higher fees, infrastructure projects demolishing black neighborhoods without compensation, black residents being denied access to mortgages (then forced to pay on installment plans where one missed payment meant eviction), tying public education to real estate taxes so poorer neighborhoods had worse-funded schools, and so on and so on.</p>
<p>The cumulative effects of discrimination over the first half of the 20th century are damning, and they have not at all been rectified.</p>
</blockquote>
<p><em>The Color of Law</em> pairs well with <a href="#negroes-and-the-gun-the-black-tradition-of-arms">Negroes and the Gun</a>.</p>
<h3 id="money-bank-credit-and-economic-cycles"><a href="https://www.goodreads.com/book/show/54388.Money_Bank_Credit_and_Economic_Cycles">Money, Bank Credit, and Economic Cycles</a></h3>
<p>This book is a dense read, and longer than 1000 pages. You could read the first 10% of the book, and be more educated than 99% of pundits blabbering away on TV.</p>
<p>The author lays out a few bold claims, and backs them up:</p>
<ul>
<li>There was a time when the treatment of bank <em>deposits</em> and bank <em>investments</em> were very different.</li>
<li>A bank making use of its customers deposits, for its own interests, was once considered simple fraud, illegal, and a banker who engaged in such fraud and was unable to make its customers whole was beheaded. (Today we call this “fractional reserve banking”)</li>
<li>Fractional-reserve banking is a legal fiction, and is actually fraud.</li>
<li>Economic cycles (“booms” followed by “busts”, or “corrections”) are assumed to be the normal state of affairs right now, but they need not exist.</li>
<li>Most of the money churning around the global economy doesn’t <em>really</em> exist, and once a very small people lose confidence in it, much of it will evaporate.</li>
</ul>
<p>It’s so good I’ve read it twice; I wish I could recommend a smaller book that gave adequate treatment to this topic, but I have not found it yet. Just read the first 10% of the book and call it a day.</p>
<p>If the topic piques your attention, but you’re not down for 1000 dense pages, <a href="https://www.goodreads.com/review/show/2028288882?book_show_action=true&amp;from_review_page=1">this review on GoodReads</a> is a comprehensive summary of the book.</p>
<h3 id="the-righteous-mind-why-good-people-are-divided-by-politics-and-religion"><a href="https://www.goodreads.com/book/show/11324722-the-righteous-mind">The Righteous Mind: Why Good People are Divided by Politics and Religion</a></h3>
<p>You know all those people who don’t agree with you?</p>
<p>They’re not evil people who hate babies and puppies and want everyone to live in squalor. They most likely want almost the exact same things you do. (Safety, security, to live in peace with their family in a community of people who doesn’t hate them, etc.)</p>
<p><em>The Righteous Mind</em> will (hopefully) give you a little compassion for those standing on the other side of a given topic. It pairs well with <a href="https://www.goodreads.com/book/show/313605.Getting_to_Yes">Getting to Yes: Negotiating an Agreement Without Giving In</a>.</p>
<p>If you find yourself resisting reading such a book, knowing that it might de-stigmatize “the other”, I humbly propose that <em>you</em> have the problem.</p>
<p>I first read this book in 2013 or so, and re-read it in 2017, and found it to be less compelling the second time through, but still, on the whole, of value.</p>
<h1 id="buildings-and-urbanism">Buildings and Urbanism</h1>
<p>To me, traditional architecture is irrelevant. Architects build buildings for other architects, not for people.</p>
<p>What <em>is</em> relevant is the buildings and environment in which we all live. We (humans) shape the environment in which we live, and we are shaped by the environment in which we live.</p>
<p>It stands to reason that an environment more conducive to use, at a lower price point, that encourages human flourishing is superior to one that is less conducive to use, costs more, and squelches human flourishing.</p>
<h3 id="how-buildings-learn"><a href="https://www.goodreads.com/book/show/38310.How_Buildings_Learn">How Buildings Learn</a></h3>
<p>This book has beautiful and fascinating photography throughout, often capturing buildings that have lasted for hundreds of years. Buildings, if allowed, change dramatically over time.</p>
<p>A theme throughout the book is that we should <em>expect</em> buildings to change, and <em>plan</em> for them to change.</p>
<p>I’m quoting a <a href="https://www.goodreads.com/review/show/44813468?book_show_action=true&amp;from_review_page=1">GoodReads reviewer</a>:</p>
<blockquote>
<p>You will love this book if, like me, you think that modern and postmodern architecture has gone terribly, terribly wrong. (Conversely, if you worship Frank Gehry, I. M. Pei, and their ilk, you will probably be offended.) Stewart Brand argues convincingly that the buildings that survive are those that can be flexible enough to adapt to the changing needs and tastes successive generations of inhabitants. He is particularly trenchant in his criticism of the overprogrammed, over-designed, sculptural architectural buildings (he calls them “magazine architecture”) that are often obsolete before they are completed, and he points out that, Frank Lloyd Wright’s opinion notwithstanding, it is not in fact a sign of architectural success if the roof leaks!</p>
<p>Also be sure to check out his very original comments on “low-road” buildings, those whose designs are so throwaway that successive inhabitants can and do feel utterly free to knock down walls, cut through floors, and otherwise jerry-rig them to adapt to current needs. It’s a brilliant exploration of an often neglected but probably ubiquitous subset of buildings.</p>
</blockquote>
<h3 id="the-death-and-life-of-great-american-cities"><a href="https://www.goodreads.com/book/show/30833.The_Death_and_Life_of_Great_American_Cities">The Death and Life of Great American Cities</a></h3>
<p>Jane Jacobs <em>loves</em> cities. She loves the unplanned, chaotic, messy way that cities evolve. She might well make you love the same thing.</p>
<p>She hits hard in this book, and her words are even more relevant today than they were in the 1950s.</p>
<p>Quoting from <a href="https://www.goodreads.com/review/show/747832578?book_show_action=true&amp;from_review_page=1">this excellent review on GoodReads</a>:</p>
<blockquote>
<p>Jacobs’s recipe for creating a healthy neighborhood has four ingredients:</p>
<ul>
<li>(1) mixed uses, so that different kinds of people are drawn to the area at different times of day for different reasons;</li>
<li>(2) a mixture of old and new buildings, so that there is low-rent space available for small businesses and low-income residents;</li>
<li>(3) small blocks, so that streets are not isolated from one another; and</li>
<li>(4) sufficient density of residents, to create the necessary amount of economic and social activity.</li>
</ul>
<p>The goal is to produce a neighborhood like her own Greenwich Village: with lots of street life, with successful residents who choose to stay long-term, with local stores and restaurants and cafes, and with a steady influx of immigrants.</p>
</blockquote>
<p>This book discusses the tension between “chaos” and “order” in a way that pairs very well with <em>Seeing like a State</em>. It pairs well with <em>Antifragile</em>, as well, as Jacobs talks about small bets, organic growth, and the dangers of “cataclysmic money” and top-down growth and planning.</p>
<h1 id="theology">Theology</h1>
<p>Every person is a functional theologian. “Theology” is defined as:</p>
<blockquote>
<p>the study of the nature of God and religious belief.</p>
</blockquote>
<p>Most of us don’t intentionally engage in “studying theology” - this would be sidestepped by someone who says “I don’t believe in god” or “maybe there is a God, maybe there isn’t, but I don’t think about it.”</p>
<p>The good news is this is an example of <em>applied theology</em>. We are all practical theologians, because we all have a stance on the existence of God. Maybe it’s apathy, maybe it’s the flying spaghetti monster.</p>
<p>Everyone has opinions on “god”. Indeed, to refuse to consider the existence of “god” is an applied theology.</p>
<p>I am one of those weird folk that have a very specific theology. Were I to have to define my theology, I’d just throw my lot in with <a href="http://matt2819.com/wsc/">the Westminster Shorter Catechism</a>.</p>
<p>If you’ve not encountered such a thing before, take a gander at <a href="https://www.thegospelcoalition.org/article/five-reasons-you-need-the-westminster-shorter-catechism/">5 reasons you need the Westminster shorter catechism</a>.</p>
<p>Its easy to get into the weeds on the finer points of theology. I argue its mostly irrelevant unless it impacts how you live your life.</p>
<p>A personal relationship with Jesus Christ is the most profound relationship you’ll ever have.</p>
<p>It can seem like a big ask, right, to say that GOD HIMSELF walked around the planet a long time ago, said stuff, did stuff, etc.</p>
<p>It seems like a big ask because it’s a really big deal if he did or not. To that end, I recommend the following book:</p>
<h3 id="the-reason-for-god-belief-in-an-age-of-skepticism"><a href="https://www.goodreads.com/book/show/1858013.The_Reason_for_God">The Reason for God: Belief in an Age of Skepticism</a></h3>
<p>This is my go-to book for someone who’s curious to know more about my crazy beliefs about the world. The author was a pastor of a church in New York City, and does a great job unpacking the core pieces of Christianity, from a range of perspectives, in a way that would lead you to agree that it is internally consistent, and is respectful to the perspectives of skeptical people.</p>
<p>By “internally consistent”, I mean that if you are comfortable with a small number of starting assumptions, the implications of those assumptions are reasonable. One of those starting assumptions (indeed, the only important one) is:</p>
<p><em>A few thousand years ago, a man named Jesus was born, said some things, was killed by the religious leaders of the day, and then didn’t stay dead.</em></p>
<p>That assumption, right there, is <em>the foundation of the entire Christian religion.</em> If it happened, it’s a really big deal. If it didn’t happen, feel pity on us all, because we are wildly misguided and stupid.</p>
<p>If that happened, the things he said have great weight. If they didn’t happen, its the ravings of a lunatic and should be forgotten posthaste.</p>
<p>I recommend the book. If you read it, shoot me an email. I’d love to hear about it.</p>
<h3 id="when-i-dont-desire-god-how-to-fight-for-joy"><a href="https://www.goodreads.com/book/show/45352.When_I_Don_t_Desire_God">When I Don’t Desire God: How to Fight for Joy</a></h3>
<p>This (and the next few books) are written by and to those who agree with the basic tenants of the Christian religion.</p>
<p>This book was written as a follow-up to <a href="https://www.goodreads.com/book/show/213367.Desiring_God">Desiring God: Meditations of a Christian Hedonist</a>.</p>
<p>I, and many others I’ve spoken about this with, often struggle to feel a sense of satisfaction, peace, and rest.</p>
<p>If you happen to care about what the Bible says, you’ll quickly recall that there is <em>no</em> satisfaction or peace apart from God. That won’t stop us all from trying to find it in something (programming! Rock climbing! Travel!) but we’ll come up short.</p>
<p>This book will point you in the right direction. It’s served me well.</p>
<h3 id="the-secret-thoughts-of-an-unlikely-convert-an-english-professors-journey-into-christian-faith"><a href="https://www.goodreads.com/book/show/14741290-the-secret-thoughts-of-an-unlikely-convert">The Secret Thoughts of an Unlikely Convert: An English Professor’s Journey Into Christian Faith</a></h3>
<p>The author of this book chronicles her extremely reluctant encounter with Christianity, and the pain and wounding that came about therein.</p>
<p>From a <a href="https://www.goodreads.com/review/show/553341153?book_show_action=true&amp;from_review_page=1">GoodReads review</a>:</p>
<blockquote>
<p>Honestly, I wasn’t expecting much. From first couple of chapters, I saw the transformation of a former feminist/lesbian English professor to Christ-embracing, sinner who was now redeemed. The chapters unpacked the clashing world views for Butterfield and how a rejection of her old self and acceptance of Christ meant the utter destruction of her life pillars. From resigning her privileged position as a college professor to faithfully engaging the roles of a pastor’s wife, Butterfield exemplified the radical nature of conversion. Throughout the book, it was clear that unlike the neatly fashioned conversion stories whispered into the mic’s in the suburban youth ministries of today, a real conversion–one wrecked by an encounter with Christ–changed lives and those around them.</p>
</blockquote>
<h3 id="extravagant-grace-gods-glory-displayed-in-our-weakness"><a href="https://www.goodreads.com/book/show/18184844-extravagant-grace">Extravagant Grace: God’s Glory Displayed in Our Weakness</a></h3>
<p>If you were raised in “the church”, chances are good you got really heavy doses of shame served to you much of your life. Shame has been my constant companion for as long as I can remember.</p>
<p>My ideal solution was always “fix myself, do everything perfectly, or change my rubric for myself so I’m living a functionally perfect life”.</p>
<p>Spoiler: this doesn’t work.</p>
<p><em>Extravagant Grace</em> is based on John Newton’s writings, and I found it to be immensely valuable to me when I read it. There are fair claims that the book leans too far in the direction of “it’s OK to wallow in your sin”, but I respectfully disagree with those claims.</p>
<p>If you are prone to heap shame upon yourself or others, this might be of value to you.</p>
<h3 id="12-ways-your-phone-is-changing-you"><a href="https://www.goodreads.com/book/show/31804439-12-ways-your-phone-is-changing-you">12 ways your phone is changing you</a></h3>
<p>From a <a href="https://www.goodreads.com/review/show/1978037057">GoodReads review</a>:</p>
<blockquote>
<p>[The author’s] writing in this book is in top form. He weaves words like Batman weaves punches. What I loved most about this book is that it shows the problem and gives a biblical solution to it WITHOUT being legalistic. He evens calls out all the technology haters and shows them why they are wrong to hate God’s good gifts just because people use them for evil.</p>
<p>This book drips joy from the very first pages. John Piper’s forward is great. Hearing the poll stats/metrics and what that means is heartbreaking. [The author’s] sections on FOMO, vices, and loneliness are some of his best. But his chapter on harshness is a must read for everyone who engages on social media, especially for people who like to involve themselves in discernment or watchdog blogs, accounts, or sites.</p>
</blockquote>
<h1 id="parenting">Parenting</h1>
<p>I’m not a parent. I’m sure once I have kids I’ll have a different set of books to recommend.</p>
<h3 id="selfish-reasons-to-have-more-kids-why-being-a-great-parent-is-less-work-and-more-fun-than-you-think"><a href="https://www.goodreads.com/book/show/10266902-selfish-reasons-to-have-more-kids">Selfish Reasons to Have More Kids: Why Being a Great Parent is Less Work and More Fun Than You Think</a></h3>
<p>This book gets the notable distinction of being why Kristi and I want to have kids in the near term. It had a profound and measurable impact on our plans and thinking. Enough said.</p>
<h3 id="the-art-of-roughhousing-good-old-fashioned-horseplay-and-why-every-kid-needs-it"><a href="https://www.goodreads.com/book/show/9861067-the-art-of-roughhousing">The Art of Roughhousing: Good Old-Fashioned Horseplay and Why Every Kid Needs It</a></h3>
<p>Physical play is deeply satisfying. I rough-housed with friends all the time when growing up. This book does make a case for the myriad benefits children, their friends, and parents can experience with roughhousing, but it’s mostly a manual for <em>how</em> to roughhouse.</p>
<p>It’s got a few dozen suggested (and illustrated) games that are appropriate for all ages, and all size differences.</p>
<blockquote>
<p>The Art of Roughhousing shows how rough-and-tumble play can nurture close connections, solve behavior problems, boost confidence, and more. Drawing inspiration from gymnastics, martial arts, ballet, traditional sports, and even animal behavior, the authors present dozens of illustrated activities for children and parents to enjoy together-everything from the “Sumo Dead Lift” to the “Rogue Dumbo.” These delightful games are fun, free, and contain many surprising health benefits for parents.</p>
</blockquote>
<p>A surprisingly practical and immediately relevant read for anyone who has kids. I plan on re-reading it once I have a child of my own.</p>
<h1 id="relationships-and-emotions">Relationships and Emotions</h1>
<p>Most of the books in this section are written by Christians, and adopt a christian perspective of relationships, which is:</p>
<blockquote>
<p>All people are broken and selfish, and it’s mostly fruitless to try to repair relationships with others and ourselves entirely independent of our relationship with God.</p>
</blockquote>
<p>(I don’t make much effort to detangle my relationships from my theology, so you might want to <a href="#theology">read that section</a> before proceeding.)</p>
<h3 id="what-did-you-expect-redeeming-the-realities-of-marriage"><a href="https://www.goodreads.com/book/show/7350849-what-did-you-expect">What Did You Expect?: Redeeming the Realities of Marriage</a></h3>
<p>This is a book written to Christians. If you’re a Christian, and married, I strongly recommend this.</p>
<p>If you’re not a Christian, go read <a href="#the-reason-for-god-belief-in-an-age-of-skepticism">The Reason for God</a> (recommended above), and then read this book.</p>
<p>Marriage is hard. Two broken sinners married each other, and live in a fallen world. Any illusions of perfection and endless honeymoon period will quickly be crushed under the reality of our own sinfulness.</p>
<p>If that seems discouraging, you’ve not yet seen how the gospel redeems <em>all</em> of this. Read the book.</p>
<h3 id="when-sinners-say-i-do-discovering-the-power-of-the-gospel-for-marriage"><a href="https://www.goodreads.com/book/show/1451719.When_Sinners_Say_I_Do_">When Sinners Say “I Do”: Discovering the Power of the Gospel for Marriage</a></h3>
<p>This is a short, digestible, practical book. Again, written for Christians.</p>
<h3 id="boundaries-when-to-say-yes-how-to-say-no-to-take-control-of-your-life"><a href="https://www.goodreads.com/book/show/944267.Boundaries">Boundaries: When to Say Yes, How to Say No to Take Control of Your Life</a></h3>
<p>Written by a Christian, but quite applicable even without adopting the author’s starting assumptions.</p>
<p>From one of the <a href="https://www.goodreads.com/questions/694330-i-want-to-read-this-book-but-i-m/answers/444361-i-m-not-a-practicing">reviewers</a>:</p>
<blockquote>
<p>I’m not a practicing Christian so I tend to skim over the biblical parts but I can’t deny that this book is super solid in terms of the wisdom it has to offer. I’m really enjoying it.</p>
</blockquote>
<p>The book’s own blurb:</p>
<blockquote>
<p>Having clear boundaries is essential to a healthy, balanced lifestyle.</p>
<p>A boundary is a personal property line that marks those things for which we are responsible.</p>
<p>In other words, boundaries define who we are and who we are not. Boundaries impact all areas of our lives:</p>
<ul>
<li>Physical boundaries help us determine who may touch us and under what circumstances</li>
<li>Mental boundaries give us the freedom to have our own thoughts and opinions</li>
<li>Emotional boundaries help us to deal with our own emotions and disengage from the harmful, manipulative emotions of others</li>
<li>Spiritual boundaries help us to distinguish God’s will from our own and give us renewed awe for our Creator</li>
</ul>
<p>Often, Christians focus so much on being loving and unselfish that they forget their own limits and limitations. When confronted with their lack of boundaries, they ask:</p>
<ul>
<li>Can I set limits and still be a loving person?</li>
<li>What are legitimate boundaries?</li>
<li>What if someone is upset or hurt by my boundaries?</li>
<li>How do I answer someone who wants my time, love, energy, or money?</li>
<li>Aren’t boundaries selfish?</li>
<li>Why do I feel guilty or afraid when I consider setting boundaries?</li>
</ul>
<p>Dr. Henry Cloud and Dr. John Townsend offer biblically-based answers to these and other tough questions, showing us how to set healthy boundaries with our parents, spouses, children, friends, co-workers, and even ourselves.</p>
</blockquote>
<h3 id="difficult-conversations-how-to-discuss-what-matters-most"><a href="https://www.goodreads.com/book/show/774088.Difficult_Conversations">Difficult Conversations: How to Discuss What Matters Most</a></h3>
<p>Not a christian book, but <em>extremely</em> relevant. The “value” of productive-but-difficult conversations is 100x almost any other kind of conversation.</p>
<p>Difficult conversations are how you define relationships, work through challenges with your significant other, resolve work conflicts, and so much more.</p>
<p>The simple act of <em>reading a book about difficult conversations</em> changes the tone of the next hard conversation you have. Instead of wanting to run from it or avoid it, you will see it as an opportunity to apply your learning, and work for a good outcome for all parties.</p>
<h3 id="daring-greatly-how-the-courage-to-be-vulnerable-transforms-the-way-we-live-love-parent-and-lead"><a href="https://www.goodreads.com/book/show/13588356-daring-greatly">Daring Greatly: How the Courage to Be Vulnerable Transforms the Way We Live, Love, Parent, and Lead</a></h3>
<p>This book is about vulnerability. Some people eschew being vulnerable. Some people try to be way too vulnerable and overshare everything with everyone.</p>
<p>Neither of those are good, but vulnerability is critical. The best way to build closeness in a relationship is (essentially) reciprocal self-disclosure.</p>
<p>Finally, vulnerability is not being “soft”, or “weak”. It has great use in simple maintenance and progression of relationships. Do you think specialized tools are appropriate to use in specialized situations? (Impact drivers in construction, version control in software, GriGris in rock climbing?)</p>
<p>Vulnerability is simply a specialized tool for specialized situations.</p>
<h3 id="search-inside-yourself-the-unexpected-path-to-achieving-success-happiness-and-world-peace"><a href="https://www.goodreads.com/book/show/12921211-search-inside-yourself">Search Inside Yourself: The Unexpected Path to Achieving Success, Happiness (And World Peace)</a></h3>
<p>Kristi and I have both become huge fans of meditation since reading this book. I seem to remember meditation getting a bad rap when I was a kid, but since then have swung around to seeing it simply as exercise for the brain.</p>
<p>If I can train my muscles for rock climbing, why shouldn’t I train my mind for focus or self-observation?</p>
<p>The general theme of this book is improving ones “Emotional Intelligence”, which happens to be a huge predictor of generalized success in working with, or being in relationships with, other people.</p>
<p>Since most of us interact with other people, doing some work to improve those interactions is appropriate.</p>
<p>A big part of that work is being able to observe how <em>you</em> are responding to certain situations.</p>
<p>Simple example: There is a world of difference between <em>being</em> angry, and <em>feeling</em> anger. The latter is superior to the former. When you <em>are</em> something, you’re controlled by it. When you <em>feel</em> something, you can choose how to respond to it.</p>
<p>This book has served Kristi and I both quite well.</p>
<h1 id="money">Money</h1>
<p>Money is… complicated. It’s easy to have too little, to have too much, or to not manage wisely the money you do have. It’s an emotionally charged topic, and there’s not a lot of good examples out there of people having a good relationship with it.</p>
<p>These two books were formative for my understanding of money (and the acquisition and spending of it).</p>
<h3 id="your-money-or-your-life"><a href="https://www.goodreads.com/book/show/78428.Your_Money_or_Your_Life">Your Money or Your Life</a></h3>
<p>From a <a href="https://www.goodreads.com/review/show/775975?book_show_action=true&amp;from_review_page=1">GoodReads review</a>:</p>
<blockquote>
<p>[Your Money or Your Life] was recommended to me by a friend, who gave up her stable teaching position to run a used bookstore after reading this book.</p>
<p>This was my first foray into the self-help genre. The prose is laughably hokey at the most inopportune times, but the message is worth slogging through the mantras and the affirmations.</p>
<p>Plus, the “nine-step program” actually works, if you’re willing to commit to it. I started out, skeptical, with a step I thought I could stick to—keeping track of my spending, and became curious about the rest of my financial health from there.</p>
<p>By the time, a year-and-a-half later, I faced the last maudlin step (calculating how much time you have left in your life), I found it so thoroughly shocking (in my case, less than half a million hours based on average life expectancy) that I realized staying in a job that made me miserable wasn’t worth it, so I quit.</p>
<p>I guess, in that sense, this book delivers on its hokey promise to change your life.</p>
</blockquote>
<h3 id="i-will-teach-you-to-be-rich"><a href="https://www.goodreads.com/book/show/11221769-i-will-teach-you-to-be-rich">I Will Teach You to be Rich</a></h3>
<p>Ramit Sethi is a polarizing, opinionated person. That said, I’ve gotten <em>jobs and raises</em> almost exclusively because of the things he’s written.</p>
<p>This book is ten years old. Some if it is outdated. Some of it his not. This book pairs well with <em>Your Money or Your Life</em> because Ramit is leans much harder towards</p>
<blockquote>
<p>if you want to spend your money on something, or anything, great! Do it! Just make sure you’ve got a system in place where you’re moving in the direction of better and better financial outcomes.</p>
</blockquote>
<p>From a <a href="https://www.goodreads.com/review/show/62830471?book_show_action=true&amp;from_review_page=1">GoodReads review</a>:</p>
<blockquote>
<p>Sethi gives advice on “automatically enabling yourself to save, invest, and spend - enjoying it, not feeling guilty…because you’re spending only what you have.”</p>
<p>His main point: automate your finances so you effortlessly save and invest, leaving you money to spend on things you love without feeling guilty. Automatic saving and investing helps overcome psychological barriers and laziness.</p>
</blockquote>
<p>You won’t find a single recommendation to track every penny you spend, or to build a budget. Most people can’t be bothered to set one up, but instead of feel guilt about this, just build a “conscious spending plan”.</p>
<h1 id="uncomfortable-books">Uncomfortable books</h1>
<p>I include this section for books that hold uncomfortable ideas for broad sections of the population. If you find yourself in total agreement with the arguments in one of these books, great! Pick another book off of this list. It’ll make you uncomfortable.</p>
<h3 id="the-problem-of-political-authority"><a href="https://www.goodreads.com/book/show/15794037-the-problem-of-political-authority">The Problem of Political Authority</a></h3>
<p>This book (its technically a textbook written by a professor from Colorado University - Boulder) argues that “political” authority cannot be arrived at from first principles, and building institutions upon a foundation of one group legally exercising coercion against another might be legal (in the same way that slavery used to be “legal”) but it’s not moral.</p>
<p><em>The Problem of Political Authority</em> is the best book I read since at least 2016.</p>
<h3 id="the-origins-of-proslavery-christianity-white-and-black-evangelicals-in-colonial-and-antebellum-virginia"><a href="https://www.goodreads.com/book/show/2507760.The_Origins_of_Proslavery_Christianity">The Origins of Proslavery Christianity: White and Black Evangelicals in Colonial and Antebellum Virginia</a></h3>
<p>When some religious authority in the US makes a pronouncement of how laws should be written, in order to sustain a moral community, I will always think of this book. There was a time when many christian thinkers expended great energy to justify the existence of slavery. So, when {topic of the day} mixed with {“political” solution to said problem} involves steamrolling over some marginalized group… I’ll just think back to the sections in this book outlining the theological justifications religious leaders used for slavery.</p>
<p>From an <a href="https://www.amazon.com/gp/customer-reviews/R25XTA19VPXJA/ref=cm_cr_dp_d_rvw_ttl?ie=UTF8&amp;ASIN=0807858773">Amazon review</a>:</p>
<blockquote>
<p>So here’s a book whose very title might alienate . . . most people. If you are not a Christian, why take the time . . . If you are . . . you don’t want to hear about it. I am still bothered by the term “Proslavery Christianity.” As a Christian and a Baptist and an American Southerner by background, it was difficult to not take some of the information presented in this book personally. I didn’t want it to be true. How can real Christians be “Proslavery?” Our American upbringing allows us to view Slavery as something separate and distinct from Religion. And our oversimplified understanding of history allows us to blame the sins of the past on long dead bad people not at all like us.</p>
<p>But what if the “bad” people were like us? What if they even thought they were right? What if they were . . . Christians.</p>
<p>Well, I don’t want to spoil the story, but . . . it happened. People, Christians included, can mix right and wrong together for so long that we end up simultaneously doing both and call them both a just cause. And we still do this. We go along with others definition of evil and good in the world. And we still allow economic, political and cultural power holders to narrow our faith to non status quo threatening endeavors.</p>
</blockquote>
<h3 id="the-privatization-of-roads-and-highways"><a href="https://www.goodreads.com/book/show/12720771-the-privatization-of-roads-and-highways">The Privatization of Roads and Highways</a></h3>
<p>That which we’ve grown up with, we (humans) tend to assume is not just ideal, but <em>morally correct</em>. We’ve grown up (as a nation) with government-built roads. It’s so impossible to imagine something else that as soon as someone suggests some crazy idea, a standard retort is “but who would build the roads?” followed by “if you like anarchy, move to Somalia!”.</p>
<p>This book addresses the first portion of that objection.</p>
<p><em>spoiler: for most of the time that humans have been building roads, they’ve been funded and built privately. Public road building has catastrophic implications for everyone.</em></p>
<p>Regrettably, the book is poorly written, <em>and</em> it’s just a collection of essays by the author, so the book covers much of the same territory multiple times. I have not found better treatment of this topic, so until then - this is the book I recommend.</p>
<h3 id="negroes-and-the-gun-the-black-tradition-of-arms"><a href="https://www.goodreads.com/book/show/17834926-negroes-and-the-gun">Negroes and the Gun: The Black Tradition of Arms</a></h3>
<p>I’m skeptical of regimes where the government is charged with protecting some minority group. My reason for this is three-fold:</p>
<ul>
<li>Governments are an expression of the people in them, who are a sub-section of the population as a whole. <em>The government tends to have the opinions of the majority of the people</em></li>
<li>If “the people” are racist, the government will be racist, <em>and will use its power to further racist goals</em>.</li>
<li>If some of the people are not racist, and try to change the government’s policies to be less racist, <em>they will not be successful until enough of the population is no longer racist that the government will, by default, be less racist.</em></li>
<li>The government’s “progressive” policies tend to lag the policies of the general population.</li>
</ul>
<p>This book outlines, among other things, how a strong federal, state, and local government used its power to inflict incredible harm on black people. Much of this harm was inflicted after the “legal” elimination of formal slavery and the Jim Crowe laws.</p>
<p>It’s rare that I cry when I read a book. This was one of maybe three books that has done that to me.</p>
<p>From a <a href="https://www.goodreads.com/review/show/1297687689?book_show_action=true&amp;from_review_page=1">GoodReads review</a>:</p>
<blockquote>
<p>A great written history of african americans from slavery through the 1970s whose only form of a self defense was a firearm.</p>
<p>This is a painful account of state and local law enforcement duplicity in the terrorizing and lynching of (mostly) southern blacks.</p>
<p>The book does a great job identifying people history has overlooked who made significant contributions to protecting their communities and eventually protecting those who were marching for equality and civil rights.</p>
<p>The book did not propose that everyone who took up arms had a satisfying outcome. In some cases, self-defense led to larger, angrier mobs of Klan and their supporters. Gun control and self defense is a difficult topic.</p>
<p>The author did a great job of presenting both sides of the argument and why the choice of firearms should be an individual choice and not a government mandate.</p>
</blockquote>
<h3 id="a-paradise-built-in-hell-the-extraordinary-communities-that-arise-in-disaster"><a href="https://www.goodreads.com/book/show/19271055-a-paradise-built-in-hell">A Paradise Built in Hell: The Extraordinary Communities That Arise in Disaster</a></h3>
<p>We’re told that when society breaks down, we devolve into a state of anarchy and will kill each other, or be killed by each other. This happens to be a self-fulfilling prophecy. It turns out that when disaster strikes, things tend to work out pretty well.</p>
<p>If we think bad things happen when disasters strike, bad things will happen.</p>
<p>If we think good things will happen when disasters strike, good things will happen. (This is related to the theory in international relations known as <a href="https://en.wikipedia.org/wiki/Constructivism_(international_relations)">constructivism</a>.</p>
<p>Since your expectations will shape your reality, if you think you might ever find yourself impacted by a disaster, this book would serve well as preparation.</p>
<p>A <a href="https://www.goodreads.com/review/show/132588792?book_show_action=true&amp;from_review_page=1">GoodReads review</a>:</p>
<blockquote>
<p>In <em>A Paradise Built in Hell</em> Solnit mounts a spirited argument that this pessimistic view of how people respond to catastrophe is fundamentally wrong. Instead, she argues, disasters are far more likely to bring out the best in people – there is a natural desire to help one another, which is actually easier to put into action, given the relaxation of social barriers that often prevails in the
wake of a disaster. You might go for years just nodding at that neighbor across the street, but after the earthquake/fire/blackout the two of you may just end up having a real conversation.</p>
<p>Solnit grounds her argument in five specific case studies:</p>
<ul>
<li>the San Francisco earthquake of 1906</li>
<li>the 1917 explosion of the munitions ship Mont Blanc in Halifax, Nova Scotia</li>
<li>Mexico City’s 1985 earthquake</li>
<li>the World Trade Center attacks of 2001</li>
<li>Hurricane Katrina and its aftermath.</li>
</ul>
</blockquote>Josh Thompsonthompsonjoshd@gmail.comI like to read, and I often recommend books to others. The recommendations below are entirely non-fiction, but have provided great value to me in very specific ways. I’ve got ~30 book recommendations grouped into following categories: Thinking Well Politics Buildings and Urbanism Theology Parenting Relationships Money Uncomfortable books I mention some books “pair well” with other books, just as certain wines pair well with a certain foods. The ideas contained within certain books may compliment (or contrast) the ideas listed in the “pairs with” book.Pry Tips and Tricks2018-05-07T13:00:00+00:002018-05-07T13:00:00+00:00https://josh.works/pry_tips_tricks<p><em>the following is cross-posted from <a href="http://development.wombatsecurity.com/development/2018/05/04/pry-tips-tricks/">development.wombatsecurity.com</a>. I wrote about some handy extra features I’ve found using Pry much of my day.</em></p>
<p>I joined the Wombat team a few months ago, and have been working on the <a href="https://www.wombatsecurity.com/products/threatsim-how-it-works">threatsim</a> product. We had a bit of a bug backlog, and myself and others have been rapidly whipping it into shape.</p>
<p><a href="https://www.wombatsecurity.com/products/threatsim-how-it-works">ThreatSim</a> is a Ruby on Rails application; any developer out there who works with Rails has probably used <a href="https://github.com/pry/pry">Pry</a> extensively in debugging their application. Pry “pauses” your application’s execution and lets you observe and manipulate state, wherever the pry happens to be.</p>
<p>Most pry usage is pretty simple - put a pry in your code, cause that line of code to be executed, and then poke around in the session in your terminal.</p>
<p>For me, this can feel unwieldy when I am trying to do a broad examination of the application. Pry is great at showing me the state of the variables contained within the method that the Pry was placed at, but I don’t always want to see just this code and its variables, I want to skip around the application and peek into different components.</p>
<h2 id="looking-at-methods">Looking at methods</h2>
<p>You can use <code class="highlighter-rouge">show-method</code> to reveal pretty much any code in your application. If you use <code class="highlighter-rouge">show-method</code> with no arguments, it will show all of the code in the method that you’ve placed the pry.</p>
<p>For example:</p>
<p><code class="highlighter-rouge">show-method</code> (with no arguments, shows current class/method location, can be similar to <code class="highlighter-rouge">whereami</code> (look at prompt) (I usually append <code class="highlighter-rouge">-l</code> to <code class="highlighter-rouge">show-method</code>, to add line numbers</p>
<!--more-->
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&lt;</span><span class="no">CookiesController</span><span class="p">:</span><span class="mh">0x00007f9156c57670</span><span class="o">&gt;</span><span class="p">:</span><span class="mi">0</span><span class="o">&gt;</span> <span class="n">show</span><span class="o">-</span><span class="nb">method</span> <span class="o">-</span><span class="n">l</span>
<span class="no">From</span><span class="p">:</span> <span class="sr">/full/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">file</span><span class="p">.</span><span class="nf">rb</span> <span class="err">@</span> <span class="n">line</span> <span class="mi">78</span><span class="p">:</span>
<span class="no">Owner</span><span class="p">:</span> <span class="no">CookiesController</span>
<span class="no">Visibility</span><span class="p">:</span> <span class="kp">private</span>
<span class="no">Number</span> <span class="n">of</span> <span class="ss">lines: </span><span class="mi">37</span>
<span class="mi">78</span><span class="p">:</span> <span class="k">def</span> <span class="nf">load_cookie_jar</span>
<span class="mi">79</span><span class="p">:</span> <span class="n">cookies</span><span class="p">[</span><span class="ss">:tasty</span><span class="p">]</span> <span class="o">=</span> <span class="kp">true</span>
<span class="mi">80</span><span class="p">:</span> <span class="n">count</span> <span class="o">=</span> <span class="n">count_cookies</span>
<span class="mi">81</span><span class="p">:</span> <span class="p">.</span>
<span class="nf">.</span>
<span class="n">lots</span> <span class="n">of</span> <span class="n">other</span> <span class="n">code</span> <span class="n">here</span>
<span class="p">.</span>
<span class="nf">.</span>
<span class="mi">111</span><span class="p">:</span> <span class="nb">require</span> <span class="s2">"pry"</span><span class="p">;</span> <span class="nb">binding</span><span class="p">.</span><span class="nf">pry</span>
<span class="mi">112</span><span class="p">:</span>
<span class="mi">113</span><span class="p">:</span> <span class="no">Repo</span><span class="o">::</span><span class="no">FakeClass</span><span class="o">::</span><span class="no">NotActuallyAModule</span><span class="p">.</span><span class="nf">do_something</span><span class="p">(</span><span class="n">options</span><span class="p">)</span>
<span class="mi">114</span><span class="p">:</span> <span class="k">end</span>
</code></pre></div></div>
<p>See that the line about to be executed (line 113)? What if you want to see what that method is, without jumping into your code editor?</p>
<p>If you want to see what that method is, it’s easy! Use <code class="highlighter-rouge">show-method</code></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">&lt;</span><span class="no">CookiesController</span><span class="p">:</span><span class="mh">0x00007f9156c57670</span><span class="o">&gt;</span><span class="p">:</span><span class="mi">0</span><span class="o">&gt;</span> <span class="n">show</span><span class="o">-</span><span class="nb">method</span> <span class="no">Repo</span><span class="o">::</span><span class="no">FakeClass</span><span class="o">::</span><span class="no">NotActuallyAModule</span><span class="p">.</span><span class="nf">do_something</span> <span class="o">-</span><span class="n">l</span>
<span class="no">From</span><span class="p">:</span> <span class="n">full</span><span class="o">/</span><span class="n">path</span><span class="o">/</span><span class="n">to</span><span class="o">/</span><span class="n">file</span><span class="p">.</span><span class="nf">rvb</span> <span class="err">@</span> <span class="n">line</span> <span class="mi">165</span><span class="p">:</span>
<span class="no">Owner</span><span class="p">:</span> <span class="c1">#&lt;Class:Repo::FakeClass::NotActuallyAModule&gt;</span>
<span class="no">Visibility</span><span class="p">:</span> <span class="kp">public</span>
<span class="no">Number</span> <span class="n">of</span> <span class="ss">lines: </span><span class="mi">22</span>
<span class="mi">165</span><span class="p">:</span> <span class="k">def</span> <span class="nf">do_something</span><span class="p">(</span><span class="n">options</span> <span class="o">=</span> <span class="p">{})</span>
<span class="mi">166</span><span class="p">:</span> <span class="n">cookies_type</span> <span class="o">=</span> <span class="n">options</span><span class="p">.</span><span class="nf">fetch</span><span class="p">(</span><span class="ss">:cookie_type</span><span class="p">)</span>
<span class="mi">167</span><span class="p">:</span> <span class="n">is_tasty?</span> <span class="o">=</span> <span class="n">options</span><span class="p">.</span><span class="nf">fetch</span><span class="p">(</span><span class="ss">:is_valid</span><span class="p">,</span> <span class="kp">false</span><span class="p">)</span>
<span class="mi">168</span><span class="p">:</span>
<span class="mi">169</span><span class="p">:</span> <span class="k">if</span> <span class="n">is_tasty?</span>
<span class="mi">170</span><span class="p">:</span> <span class="n">log_it</span> <span class="s2">"load_cookies"</span><span class="p">,</span> <span class="s2">"is_tasty"</span>
<span class="mi">171</span><span class="p">:</span> <span class="n">options</span><span class="p">[</span><span class="ss">:consumed</span><span class="p">]</span> <span class="o">=</span> <span class="kp">nil</span>
<span class="mi">172</span><span class="p">:</span> <span class="n">options</span><span class="p">[</span><span class="ss">:pairs_with</span><span class="p">]</span> <span class="o">=</span> <span class="s2">""</span>
<span class="mi">173</span><span class="p">:</span>
<span class="mi">174</span><span class="p">:</span>
<span class="p">.</span>
<span class="nf">.</span>
<span class="o">.</span>
<span class="mi">185</span><span class="p">:</span> <span class="k">end</span>
<span class="mi">186</span><span class="p">:</span> <span class="k">end</span>
</code></pre></div></div>
<p>(I’m still adding <code class="highlighter-rouge">-l</code> to force line-numbers to be printed out.</p>
<p><em><a href="https://github.com/pry/pry#code-browsing">more about <code class="highlighter-rouge">show-source</code> from Pry</a></em></p>
<h2 id="where-was-i">Where was I?</h2>
<p>Sometimes, I go so far down a rabbit hole of digging around in Pry, I forget where the <code class="highlighter-rouge">binding.pry</code> actually is, and what I was trying to do in the first place. (Or I have a few conditional <code class="highlighter-rouge">binding.pry</code> statements, and I don’t recall which one got hit)</p>
<p>Enter <code class="highlighter-rouge">whereami</code></p>
<p>This command simply prints out the code surrounding the current <code class="highlighter-rouge">binding.pry</code>. It’s run by default as soon as you hit the pry, which is how you can quickly get your bearings.</p>
<p><em><a href="https://github.com/pry/pry/wiki/Runtime-invocation#Whereami">from the pry docs: whereami</a></em></p>
<h2 id="view-stack-traces">View stack traces</h2>
<p>What was that stack trace from the last exception you saw?</p>
<p><code class="highlighter-rouge">wtf</code> puts said stack trace:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&lt;CookiesController:0x00007f9156c57670&gt;:0&gt; not_a_variable
NameError: undefined <span class="nb">local </span>variable or method <span class="sb">`</span>not_a_variable<span class="s1">' for #&lt;CookiesController:0x00007f9156c57670&gt;
from (pry):16:in `load_cookie_jar'</span>
</code></pre></div></div>
<p>and then, anytime later, call <code class="highlighter-rouge">wtf</code> in pry:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>&lt;CookiesController:0x00007f9156c57670&gt;:0&gt; wtf
Exception: NameError: undefined <span class="nb">local </span>variable or method <span class="sb">`</span>not_a_variable<span class="s1">' for #&lt;CookiesController:0x00007f9156c57670&gt;
--
0: (pry):16:in `load_live_action'</span>
1: /Users/joshthompson/.rvm/gems/ruby-2.3.7/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in <span class="sb">`</span><span class="nb">eval</span><span class="s1">'
2: /Users/joshthompson/.rvm/gems/ruby-2.3.7/gems/pry-0.10.3/lib/pry/pry_instance.rb:355:in `evaluate_ruby'</span>
3: /Users/joshthompson/.rvm/gems/ruby-2.3.7/gems/pry-0.10.3/lib/pry/pry_instance.rb:323:in <span class="sb">`</span>handle_line<span class="s1">'
4: /Users/joshthompson/.rvm/gems/ruby-2.3.7/gems/pry-0.10.3/lib/pry/pry_instance.rb:243:in `block (2 levels) in eval'</span>
</code></pre></div></div>
<p><em><a href="https://github.com/pry/pry/wiki/Exceptions#wtf">more about <code class="highlighter-rouge">wtf</code> in Pry</a></em></p>
<h2 id="breakpoints-in-pry">Breakpoints in Pry</h2>
<p>What about when looking around the state of your application, you decide you want to examine in pry a different method?</p>
<p>Using <code class="highlighter-rouge">show-source &lt;method_name&gt;</code> doesn’t let you interact with the method.</p>
<p>Enter breakpoints. Just like with javascript in the browser, you can add/remove breakpoints to your code with Pry. You don’t have to exit the session, jump to the new method, and add a <code class="highlighter-rouge">binding.pry</code> to it.</p>
<p>You’ll need to add <a href="https://github.com/deivid-rodriguez/pry-byebug">pry-byebug</a> to your Gemfile.</p>
<p>With Pry-byebug, breakpoint functionality is fairly straightforward:</p>
<ul>
<li><code class="highlighter-rouge">break</code> shows all current breakpoints. (this list should be empty if you’re running <code class="highlighter-rouge">break</code> for the first time.</li>
<li><code class="highlighter-rouge">break &lt;Class#method&gt;</code> adds a breakpoint to the start of the given method.</li>
</ul>
<p>if you add a breakpoint, and call <code class="highlighter-rouge">break</code> you’ll see something like:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="c"># Enabled At</span>
<span class="nt">-------------</span>
1 Yes Threatsim::LandingPage::GuidLoader.find_guid
</code></pre></div></div>
<p>Remove the first breakpoint with <code class="highlighter-rouge">break --delete 1</code></p>
<p><code class="highlighter-rouge">break --help</code> is a fruitful summary of what breakpoint-related methods are available to you.</p>
<p><em><a href="https://github.com/deivid-rodriguez/pry-byebug#breakpoints">more about breakpoints in Pry</a></em></p>
<h2 id="calling-all-callers">Calling all callers</h2>
<p>Ever wanted to see what called the code that hit your breakpoint?</p>
<p>I sure have!</p>
<p>As usual, Stack Overflow has <a href="https://stackoverflow.com/a/21620257/3210178">a most helpful answer</a>.</p>
<p>You can just call <code class="highlighter-rouge">caller</code> in pry, to get a full list everything involved in the current stack.</p>
<p>The author of the post points out that you immediately get a giant array of mostly irrelevant items, and suggests filtering by keyword, using a one-liner like so:</p>
<p><code class="highlighter-rouge">caller.select {|line| line.include? "current_repo_name" }</code></p>
<p>Or, alternatively:</p>
<p><code class="highlighter-rouge">caller.reject { |l| l[".rvm/gems"] }</code></p>
<h3 id="additional-reading-or-articles-that-helped-me-learn-more-about-pry">Additional reading, or articles that helped me learn more about Pry:</h3>
<ul>
<li><a href="http://jonathan-jackson.net/2012/05/03/pry-session-102">Pry 102: Advanced Features</a></li>
<li><a href="https://gist.github.com/lfender6445/9919357">Pry Cheat Sheet</a></li>
</ul>Josh Thompsonthompsonjoshd@gmail.comthe following is cross-posted from development.wombatsecurity.com. I wrote about some handy extra features I’ve found using Pry much of my day. I joined the Wombat team a few months ago, and have been working on the threatsim product. We had a bit of a bug backlog, and myself and others have been rapidly whipping it into shape. ThreatSim is a Ruby on Rails application; any developer out there who works with Rails has probably used Pry extensively in debugging their application. Pry “pauses” your application’s execution and lets you observe and manipulate state, wherever the pry happens to be. Most pry usage is pretty simple - put a pry in your code, cause that line of code to be executed, and then poke around in the session in your terminal. For me, this can feel unwieldy when I am trying to do a broad examination of the application. Pry is great at showing me the state of the variables contained within the method that the Pry was placed at, but I don’t always want to see just this code and its variables, I want to skip around the application and peek into different components. Looking at methods You can use show-method to reveal pretty much any code in your application. If you use show-method with no arguments, it will show all of the code in the method that you’ve placed the pry. For example: show-method (with no arguments, shows current class/method location, can be similar to whereami (look at prompt) (I usually append -l to show-method, to add line numbersKeyboard Shortcut to Toggle Bookmarks Bar in Firefox2018-02-15T13:00:00+00:002018-02-15T13:00:00+00:00https://josh.works/toggle_bookmarks_bar_in_firefox<p>A few weeks ago, after <a href="https://www.mozilla.org/en-US/firefox/">Firefox Quantum</a> came out, I decided to give it a go.</p>
<p>Turns out, Firefox is great! It was a near-seamless transition, and Firefox has a much lower memory footprint, as well as features Chrome does not have, like <a href="https://support.mozilla.org/en-US/kb/tracking-protection">Tracking Protection</a> and <a href="https://blog.mozilla.org/firefox/reader-view/">Reader View</a></p>
<p>But something was bothering me to no end. I could <em>not</em> find a keyboard shortcut to toggle the visibility of the bookmarks toolbar.</p>
<p>I take a lot of screenshots throughout the day, and share them within my company. I don’t really want my bookmarks bar taking up space in the screenshot, but I do sometimes need it to find actual bookmarks.</p>
<p>In Chrome, <code class="highlighter-rouge">Cmd-Shift-B</code> toggles the bookmarks bar visibility. In Firefox, that combo shows your history.</p>
<!--more-->
<p>The primary fix is to use the Mac <em>operating system</em> to set an App-specific keyboard shortcut. This is bananas, and I’ve never done it before.</p>
<p>Here’s how to do it:</p>
<p>Navigate to <code class="highlighter-rouge">System Preferences &gt; Keyboard</code>. From the options on the top bar (Keyboard, Text, Shortcuts, etc) select <code class="highlighter-rouge">Shortcuts</code>.</p>
<p>In the left-hand sidebar, choose <code class="highlighter-rouge">App Shortcuts</code>.</p>
<p><img src="/images/2018-02-15_01.jpg" alt="find the app shortcut screen" /></p>
<p>Hit the <code class="highlighter-rouge">+</code> icon to add a new shortcut, and select <code class="highlighter-rouge">Firefox.app</code> from the list of applications.</p>
<p>Enter this string exactly where in the Menu Title box:</p>
<p><code class="highlighter-rouge">View-&gt;Toolbars-&gt;Bookmarks Toolbar</code></p>
<p><img src="/images/2018-02-15_02.jpg" alt="add the shortcut" /></p>
<p>I used <code class="highlighter-rouge">Cmd-Shift-Y</code> as my shortcut - it took a few attempts to find something that didn’t conflict with existing OS/Firefox shortcuts.</p>
<p>(And yes, I tried to unmap <code class="highlighter-rouge">Cmd-Shift-B</code>, to free it up for this shortcut, and could not. If you figure out how, please let me know.)</p>
<h2 id="the-result">The result</h2>
<p><img src="/images/2018-02-15_03.gif" alt="toggle away, friend" /></p>
<h2 id="firefoxs-gotcha">Firefox’s Gotcha</h2>
<p>Sometimes the keyboard shortcut doesn’t work. It seems like Firefox “forgets” what it’s supposed to do.</p>
<p>If that happens, I’m able to set things back as they should be by toggling the bookmarks bar manually, once, and then the shortcut works again.</p>
<p>(This setting lives in <code class="highlighter-rouge">View &gt; Toolbars &gt; Bookmarks Toolbar</code>)</p>
<h3 id="additional-readinguseful-resources">Additional Reading/Useful resources</h3>
<ul>
<li><a href="http://osxdaily.com/2017/08/08/create-custom-keyboard-shortcut-mac/">OSX Daily: How to Create Custom Keyboard Shortcuts in Mac OS</a></li>
<li><a href="https://support.mozilla.org/en-US/questions/865261">Mozilla Support Forum: Is there a short/hotkey to show/hide bookmark toolbar?</a></li>
</ul>Josh Thompsonthompsonjoshd@gmail.comA few weeks ago, after Firefox Quantum came out, I decided to give it a go. Turns out, Firefox is great! It was a near-seamless transition, and Firefox has a much lower memory footprint, as well as features Chrome does not have, like Tracking Protection and Reader View But something was bothering me to no end. I could not find a keyboard shortcut to toggle the visibility of the bookmarks toolbar. I take a lot of screenshots throughout the day, and share them within my company. I don’t really want my bookmarks bar taking up space in the screenshot, but I do sometimes need it to find actual bookmarks. In Chrome, Cmd-Shift-B toggles the bookmarks bar visibility. In Firefox, that combo shows your history.