StopDropAndrew2011-03-21T20:15:15-07:00http://www.stopdropandrew.com/Andrew Grimstopdropandrew@gmail.comMaking image juice: squeezing the unused bytes out of your images2011-03-21T00:00:00-07:00http://www.stopdropandrew.com//2011/03/21/making-image-juice-squeezing-unused-bytes-out-of-images<p><img src="/images/squeeze_pop.jpg"></p>
<p>Image size is definitely a case of &#8220;less is more.&#8221; Many images on the web are larger than they need to be, and there has been a recent surge in the movement to strip out excess bits while maintaining a lossless image. <a href="http://www.smushit.com/ysmush.it/">Smush.it™</a> is a great example. Unfortunately, even though it runs on open source image utilities, there is no <span class="caps">API</span> and <a href="http://www.yahoo.com">Yahoo!</a> seems to have no plan to release an open source version.</p>
<p>Have no fear! It isn&#8217;t doing anything complicated, so why not just do it ourselves. Plus, the authors basically laid out what they do in the excellent <a href="http://www.amazon.com/Even-Faster-Web-Sites-Performance/dp/0596522304">Even Faster Web Sites</a>.</p>
<h3>Image Squeeze</h3>
<p><a href="http://www.github.com/stopdropandrew/image_squeeze">Image Squeeze</a> is a pluggable library for optimizing images. The only library it requires is ImageMagick (specifically it uses identify to figure out what type of image it&#8217;s looking at). The default processors included with the library require pngcrush, jpegtran, gifsicle (for animated gifs), and ImageMagick&#8217;s convert. Here&#8217;s a quick example:</p>
<ul>
<li>set up an ImageSqueeze with the default processors<br />
<pre>squeezer = ImageSqueeze.new(:default_processors =&gt; true)</pre></li>
</ul>
<ul>
<li>in-place squeeze of our png<br />
<pre>squeezer.squeeze!(&#8216;my_logo.png&#8217;)</pre></li>
</ul>
<p>If you don&#8217;t want to install one of those for some reason, don&#8217;t worry, image_squeeze just won&#8217;t include the processor if the utility isn&#8217;t available.</p>
<h3>Adding your own processors</h3>
<p>Let&#8217;s say PngCrush isn&#8217;t good enough for you, and you want to add an OptiPNG processor as well. Easy enough, we can just plug that in.</p>
<script src="http://gist.github.com/421846.js?file=OptiPNGProcessor"></script><p>So you just need to implement two methods, once which returns an image type (constants defined in ImageSqueeze) and a squeeze method (takes an input filename and leaves a new image in the output filename). <strong>Note:</strong> The default processors are all lossless quality, if you implement your own it&#8217;s up to you to verify that you aren&#8217;t losing quality.</p>
<p>Now let&#8217;s load it in with our defaults,</p>
<pre>squeezer = ImageSqueeze.new(:default_processors =&gt; true, :processors =&gt; [ImageSqueeze::OptiPNGProcessor])</pre>
<p>Or if you don&#8217;t like the defaults, just use your own processors</p>
<pre>squeezer = ImageSqueeze.new(:processors =&gt; [ImageSqueeze::OptiPNGProcessor])</pre>
<p>That was pretty useful, and OptiPNG is great, so I just added it to the defaults. Progress!</p>
<p>Now just hook this into wherever you handle uploaded images and you&#8217;re set. I&#8217;d definitely recommend doing this out-of-process as far as the server request, or you&#8217;ll kill your request times, but I&#8217;ll leave that up to you.</p>
<p>One thing I&#8217;d really like to do is make a pre-commit hook that process any images you are checking in so you don&#8217;t have to worry about it. But that&#8217;s for another time&#8230;</p>
<p>Woohoo less bytes!</p>Where Unicorns go to die: Watching unicorn workers with monit2010-06-01T00:00:00-07:00http://www.stopdropandrew.com//2010/06/01/where-unicorns-go-to-die-watching-unicorn-workers-with-monit<p><img src="/images/dead-unicorn1.jpg"></p>
<p>While working on switching Kongregate from Mongrel to Unicorn I was googling around for a monit recipe for watching Unicorn child processes and unfortunately couldn&#8217;t find anything. The real issue is that monit will only watch either just the parent process or a the entire process family (parent including all of its children). What I really needed was the ability to watch each child process individually, and kill them when their memory usage grows out of control.</p>
<p>I was able to find another process monitoring tool called <a href="http://github.com/arya/bluepill">bluepill</a> which looks like it was built almost explicitly for this reason. While monit can be a pain in the ass, it&#8217;s stable and I don&#8217;t have any other reason to switch. Ideally monit will eventually allow monitoring child process, until then I found a decent workaround.</p>
<h3>The problem</h3>
<p>Since the ideal case of monit watching child processes doesn&#8217;t exist, why not have monit just watch each child process individually. Well the odd thing here is that monit wants to start and stop each process, but the Unicorn master starts each worker, so we&#8217;ll have monit just worry about killing the children.</p>
<h3>init.d script</h3>
<p>For my init.d script, I use the script that is packaged with the Unicorn source with a simple change, I add a kill_worker method.</p>
<script src="http://gist.github.com/421663.js?file=kill+worker"></script><h3>Unicorn master monit config</h3>
<p>This is the simple part, but we want monit to watch the master worker, it shouldn&#8217;t ever grow about our memory limit, but we do want monit to make sure it&#8217;s running.</p>
<script src="http://gist.github.com/421670.js?file=gistfile1.txt"></script><h3>Unicorn workers monit config</h3>
<p>Here&#8217;s where the magic happens&#8230;I guess. So the weird part about monit watching Unicorns is that monit wants to start each process it watches. We&#8217;ll just trick monit into doing a no-op for the start and let it watch the pid files to figure out when the master starts the workers.</p>
<script src="http://gist.github.com/421674.js?file=Unicorn+worker+monit+config"></script><h3>Unicorn post_fork</h3>
<p>The final piece of the puzzle is making each child write out its own pid file, which we do with a simple after_fork hook. I use port for the pid file, for slightly easier debugging, but there is no real reason for it.</p>
<script src="http://gist.github.com/421683.js?file=Unicorn+config"></script><p>In the end the final solution is kind of cheesy, since monit just keeps doing nothing until the master starts the workers, but once everything gets started up you are golden.</p>
<p>Finally, here&#8217;s the monit commands to start everything up.</p>
<pre>monit start all -g unicorn
monit start all -g unicorn_workers</pre>Finding ids and other column values fast with ActiveRecord2010-01-28T00:00:00-08:00http://www.stopdropandrew.com//2010/01/28/finding-ids-fast-with-active-record<p>Chances are at some point you&#8217;ve wanted to get the values of a single column from ActiveRecord. You might have done something like this:</p>
<div class='highlight'><pre><code class='ruby'><span class='n'>user_ids</span> <span class='o'>=</span> <span class='no'>User</span><span class='o'>.</span><span class='n'>find</span><span class='p'>(</span><span class='ss'>:all</span><span class='p'>)</span><span class='o'>.</span><span class='n'>map</span><span class='p'>(</span><span class='o'>&amp;</span><span class='ss'>:id</span><span class='p'>)</span>
</code></pre>
</div>
<p>That&#8217;s great and all, but hopefully somebody points out that you can pass ActiveRecord#find a :select key.</p>
<div class='highlight'><pre><code class='ruby'><span class='n'>user_ids</span> <span class='o'>=</span> <span class='no'>User</span><span class='o'>.</span><span class='n'>find</span><span class='p'>(</span><span class='ss'>:all</span><span class='p'>,</span> <span class='ss'>:select</span> <span class='o'>=&gt;</span> <span class='s1'>&#39;id&#39;</span><span class='p'>)</span><span class='o'>.</span><span class='n'>map</span><span class='p'>(</span><span class='o'>&amp;</span><span class='ss'>:id</span><span class='p'>)</span>
</code></pre>
</div>
<p>Now we&#8217;re getting somewhere. On a &#8220;wide&#8221; table (ie. one with a lot of columns, large columns, or binary types stored separately) this alone can be a huge improvement, as just sending all that data across the wire can really waste time. While ActiveRecord is generally good enough performance wise, sometimes getting a little closer to the metal is appropriate, and in comes find_ids.</p>
<div class='highlight'><pre><code class='ruby'><span class='n'>user_ids</span> <span class='o'>=</span> <span class='no'>User</span><span class='o'>.</span><span class='n'>find_ids</span>
</code></pre>
</div>
<h3 id='benchmarks'>Benchmarks</h3>
<p>Here&#8217;s some <a href='http://github.com/stopdropandrew/find_ids/blob/master/benchmarks/benchmarks.rb'>benchmarks</a> against a fairly &#8220;wide&#8221; table with 100 records, 10,000 times for each style:</p>
<pre><code> user system total real
find(:all).map 26.460000 3.130000 29.590000 ( 72.656614)
find(:all, :select).map 12.770000 0.560000 13.330000 ( 16.232474)
find_ids 7.870000 0.340000 8.210000 ( 10.465719)</code></pre>
<h3 id='examples'>Examples</h3>
<p>But that&#8217;s not all, it also natively supports other columns, associations, and named scopes.</p>
<div class='highlight'><pre><code class='ruby'><span class='n'>emails</span> <span class='o'>=</span> <span class='no'>User</span><span class='o'>.</span><span class='n'>find_column_values</span><span class='p'>(</span><span class='s1'>&#39;email&#39;</span><span class='p'>)</span>
<span class='no'>User</span><span class='o'>.</span><span class='n'>wristbands</span><span class='o'>.</span><span class='n'>find_ids</span>
<span class='no'>User</span><span class='o'>.</span><span class='n'>scoped</span><span class='p'>(</span><span class='ss'>:order</span> <span class='o'>=&gt;</span> <span class='s1'>&#39;created_at desc&#39;</span><span class='p'>)</span><span class='o'>.</span><span class='n'>find_ids</span>
</code></pre>
</div>
<h3 id='installation'>Installation</h3>
<p>script/plugin install git://github.com/stopdropandrew/find_ids.git</p>
<h3 id='issues'>Issues</h3>
<p>It currently only works with MySQL, but submissions for other adapters are <a href='http://www.github.com/stopdropandrew/find_ids'>welcome</a>.</p>Making an authoritative Git repository automatically push to Github2010-01-20T00:00:00-08:00http://www.stopdropandrew.com//2010/01/20/making-an-authoritative-git-repo-auto-push-to-github<p>I&#8217;m still in the process of learning Git, and while it&#8217;s been fairly rough in the beginning, I feel like I&#8217;m starting to round the curve of really being productive. While we are still primarily using Subversion at <a href="http://www.kongregate.com">Kongregate</a>, I&#8217;ve been trying to use Git whenever possible.</p>
<p>For a side project I&#8217;ve been working on, I wanted to have my authoritative Git repo in our colo, behind our <span class="caps">VPN</span>, to both speed up deploys and to know that we don&#8217;t need to rely on a third party service. But, at the same time, I wanted to be able to use Github&#8217;s awesome UI.</p>
<h3>hooks/post-receive</h3>
<p>There&#8217;s not a lot to it, but just set up Github as a remote for your server side repo, then drop this into your hooks/post-receive and be sure to make it executable:</p>
<div class="highlight"><pre><code class="ruby"><span class="c1">#!/usr/bin/env ruby</span>
<span class="no">STDIN</span><span class="o">.</span><span class="n">read</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&quot;</span><span class="se">\n</span><span class="s2">&quot;</span><span class="p">)</span><span class="o">.</span><span class="n">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">line</span><span class="o">|</span>
<span class="n">oldrev</span><span class="p">,</span> <span class="n">newrev</span><span class="p">,</span> <span class="n">refname</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">&#39; &#39;</span><span class="p">)</span>
<span class="k">if</span> <span class="n">refname</span><span class="o">.</span><span class="n">match</span><span class="p">(</span><span class="sr">/^refs\/heads\/(.*)/</span><span class="p">)</span>
<span class="n">branch</span> <span class="o">=</span> <span class="vg">$1</span>
<span class="sb">`git push origin </span><span class="si">#{</span><span class="n">branch</span><span class="si">}</span><span class="sb">`</span>
<span class="k">else</span>
<span class="nb">puts</span> <span class="s2">&quot;</span><span class="si">#{</span><span class="n">refname</span><span class="si">}</span><span class="s2"> was weird, not sure what to do.&quot;</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Nothing too special, but I was pretty happy to get it all working without too much effort.</p>