Tyler Butler Feedhttp://www.tylerbutler.comThe 5 most recent posts from http://www.tylerbutler.com.Wed, 11 Feb 2015 00:00:00 -0000Documentation, Versions, and Read the Docshttp://www.tylerbutler.com/2015/02/documentation-versions-and-read-the-docs/ <p>Engineer is a side project for me right now. That means that while I am actively working on Engineer pretty regularly, releases themselves are not necessarily regular. I&#8217;ve adopted a repository/branch structure that&#8217;s influenced by <a href="http://nvie.com/posts/a-successful-git-branching-model/">git flow</a>, so my main development branch is not <em>master</em>, it&#8217;s <em>dev</em>.</p>
<ul>
<li><em>master</em>: The most recent released version of the&nbsp;code</li>
<li><em>dev</em>: The in-development version of the&nbsp;code</li>
</ul>
<p>Up until <a href="/2014/05/engineer-v0-5-0-released/">Engineer v0.5.0</a>, when you went to GitHub, you saw <em>master</em> by&nbsp;default.</p>
<p>When you go to GitHub, I want you to see the latest in-development version. The reason is pretty simple: Since official releases are fairly slow, but I actually make changes fairly often, I want to make sure that activity is shown on the GitHub homepage &#8211; via the &#8216;x days ago&#8217; text that shows up on the far right of the code listing. My hypothesis is that people make some judgements based on the activity level of project. If people are searching for a static site generator, and they come across Engineer, I don&#8217;t want them to think that the project is abandoned and simply leave. If my default branch shown in GitHub is <em>master</em>, then it looks as though the project isn&#8217;t under active development at first glance, which clearly isn&#8217;t what I want. Thus, I want GitHub to display the <em>dev</em> branch by default, which is easy enough to change in the repository settings. I made this change along with the release of Engineer v0.5.0, so now when you go to the repository on GitHub, you&#8217;ll see the <em>dev</em> branch by&nbsp;default.</p>
<p>There&#8217;s still a problem, though, related to the fact that I host <a href="https://engineer.readthedocs.org/">Engineer&#8217;s public documentation</a> on <a href="https://readthedocs.org/">Read the Docs</a> (<span class="caps">RTD</span>). Imagine someone finds my project on GitHub, likes what they see, and installs the release version using&nbsp;pip:</p>
<div class="codehilite"><pre>pip install engineer
</pre></div>
<p>Seems to be the natural thing to do, right? If they click the link in Engineer&#8217;s <span class="caps">README</span> to visit the docs at <span class="caps">RTD</span>, then I want them to go to the version of the docs that corresponds to the version they just installed &#8211; the most recent released&nbsp;version.</p>
<p>Now this is also relatively easy to do. <span class="caps">RTD</span> actually has some smarts around <a href="https://docs.readthedocs.org/en/latest/versions.html">multiple versions of documentation</a>. <span class="caps">RTD</span> offers a few different settings that are relevant to my&nbsp;goals.</p>
<p>First, there is a baked in &#8216;version&#8217; identifier called <em>latest</em> which is intended to point to the most recent version of your&nbsp;docs:</p>
<blockquote>
<p>In the normal case, the latest version will always point to the most up to date development code. If you develop on a branch that is different than the default for your <span class="caps">VCS</span>, you should set the <strong>Default Branch</strong> to that&nbsp;branch.</p>
</blockquote>
<p>Of course, in my case, development is done on the <em>dev</em> branch, so I want <em>latest</em> to point to that branch. Fortunately that&#8217;s easy to change, as the second sentence above alludes to. In the <em>Advanced Settings</em> section of the <span class="caps">RTD</span> dashboard, you&#8217;ll find a <em>Default branch</em> setting, in which I entered <em>dev</em>. Great; now <em>latest</em> points to <em>dev</em>.</p>
<p>The second setting of relevance in <span class="caps">RTD</span> is the <em>default version</em>. This controls what version of your docs <code>/</code> redirects to. By default this will be <em>latest</em>, but since I want <code>/</code> to always redirect to the most recent <em>released version</em> of Engineer, I changed this to <em>master</em>. Cool; now <code>/</code> simply redirects to the version of my docs from the <em>master</em> branch, which will always be the most recent released&nbsp;version.</p>
<p>There is, of course, still a problem. Ideally, I would like links that people follow to go to the version of the documentation that matches the version of the code they&#8217;re coming from and vice-versa. In other words, I would like the documentation link from the <span class="caps">README</span> file in the <em>master</em> branch to go to https://engineer.readthedocs.org/en/master/, and the link in the <em>dev</em> branch to go to&nbsp;https://engineer.readthedocs.org/en/latest/.</p>
<p>Unfortunately, that&#8217;s not really possible. Sure, I could build some intelligent redirector or something that would look at the referrer <span class="caps">URL</span> and redirect to the appropriate docs version, but that&#8217;s not something I want to build anytime soon. The best I can do is provide some notes in the documentation itself telling people that they <em>may</em> be looking for a version of documentation that is different from what they&#8217;re seeing. It&#8217;s not quite ideal from my perspective, but I think it&nbsp;helps.</p>
<p>So bottom line, this is what I&#8217;ve wound up&nbsp;with:</p>
<ul>
<li>If you go directly to https://engineer.readthedocs.org/, you&#8217;ll get the latest <em>released</em> version of the docs, which will correspond to what most people will install using&nbsp;pip.</li>
<li>If you visit the <a href="https://github.com/tylerbutler/engineer">GitHub repo</a>, you&#8217;ll see the most recent in-development version of the code. The <span class="caps">README</span> links to https://engineer.readthedocs.org/, which as I mentioned earlier will redirect to the <em>released</em> version of the&nbsp;docs.</li>
<li>The docs themselves contain notes redirecting people to https://engineer.readthedocs.org/en/latest/ if they need the most recent version of the docs. <span class="caps">RTD</span> itself also contains links to all versions of the docs, but I don&#8217;t think most people know that and if they do, it may not be clear which version they&nbsp;want.</li>
</ul>
Wed, 11 Feb 2015 00:00:00 -0000http://www.tylerbutler.com/2015/02/documentation-versions-and-read-the-docs/Loot Tables →http://www.lostgarden.com/2014/12/loot-drop-tables.html <p>Daniel Cook tells you <a href="http://www.lostgarden.com/2014/12/loot-drop-tables.html">more than you ever needed to know about loot tables</a>. Unless you&#8217;re a game designer, of&nbsp;course&#8230;</p>
Fri, 09 Jan 2015 00:00:00 -0000http://www.tylerbutler.com/2015/01/loot-tables/Installing Binary Python Packages on Windowshttp://www.tylerbutler.com/2014/12/installing-binary-python-packages-on-windows/ <p><em>Updated December 31,&nbsp;2014</em></p>
<p>In my <a href="/2012/05/how-to-install-python-pip-and-virtualenv-on-windows-with-powershell/">Python Windows installation guide</a>, I concluded with the following&nbsp;paragraph:</p>
<blockquote>
<p>After all of that’s done you should be good to go! You can pop open a PowerShell window and create/switch to virtualenvs as needed and install packages using pip. At this point you should have most of what you need to follow the installation instructions for most Python packages (except those that require C extension compilation, but that’s a topic for another&nbsp;post).</p>
</blockquote>
<p>Despite writing the initial version of that guide over two years ago, I never got around to writing that &#8216;other post&#8217; to cover installing packages that require C extension compilation. I personally rarely run into this need, but when it comes up it&#8217;s incredibly annoying. And guess what? It came up recently when I tried to install <a href="https://www.samba.org/~jelmer/dulwich/">Dulwich</a>, a Python implementation of Git. Fortunately for you, I decided to take this opportunity to actually write the&nbsp;guide.</p>
<h2>Do I Need This&nbsp;Guide?</h2>
<p>You only <em>need</em> this guide if you try to install a Python package on Windows and you get an error like&nbsp;this:</p>
<div class="codehilite"><pre>building &#39;Crypto.Random.OSRNG.winrandom&#39; extension
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.
error: Unable to find vcvarsall.bat
</pre></div>
<p>Ahhh, the dreaded <em>Unable to find vcvarsall.bat</em> error&#8230; This error means that the package you&#8217;re installing has a C extension that needs to be compiled. Python itself is compiled using a specific version of the Visual Studio C++ compiler, and when you try to install packages that require C compilation, it goes looking for the compiler locally so it can compile the necessary&nbsp;stuff.</p>
<p>Of course, in your case, you probably don&#8217;t have Visual Studio, or if you do, it&#8217;s not the right version, or you don&#8217;t have the C++ compiler installed, or it&#8217;s installed in a non-standard location, or a specific environment variable isn&#8217;t set, or you forgot to reopen your PowerShell/cmd window after you set that environment variable&#8230; As you can see, there are many many reasons why this is painful. Don&#8217;t worry; you&#8217;re not alone. As <a href="http://stackoverflow.com/questions/2817869/error-unable-to-find-vcvarsall-bat">this question on Stack Overflow</a> indicates, lots of people run into this problem, and there are lots of ways to &#8216;solve&#8217; it.<sup id="fnref:1"><a class="footnote-ref" href="#fn:1" rel="footnote">1</a></sup></p>
<p>Because of the relative complexity of the problem, and all the potential ways various solutions could be thwarted, this used to be annoyingly difficult to address, but now it&#8217;s pretty&nbsp;easy.</p>
<h2>Installing the Microsoft C++ Compiler for Python&nbsp;2.7</h2>
<p>I am extremely glad I didn&#8217;t try to write this guide a few years ago, when I wrote my Python installation guide, because in the time since then, some smart person<sup id="fnref:2"><a class="footnote-ref" href="#fn:2" rel="footnote">2</a></sup> at Microsoft felt the collective pain and anguish of Python developers everywhere and made <a href="https://www.microsoft.com/en-us/download/details.aspx?id=44266">a package available directly on microsoft.com</a> that &#8220;contains the compiler and set of system headers necessary for producing binary wheels for Python 2.7 packages.&#8221; Hooray!<sup id="fnref:3"><a class="footnote-ref" href="#fn:3" rel="footnote">3</a></sup></p>
<p>If you <a href="https://www.microsoft.com/en-us/download/details.aspx?id=44266">download that package</a> and install it, you should be able to successfully install whatever package that was erroring out with <em>Unable to find vcvarsall.bat</em> before. Make sure you re-open any PowerShell or cmd windows you had open to make sure your environment variables are up to date. Oh, and in case you care, the compiler and all its supporting files can be found in <code>~\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0</code> after&nbsp;installation.</p>
<p>If you&#8217;re still having problems, chances are your version of setuptools is out of&nbsp;date.</p>
<h2>Update setuptools and&nbsp;pip</h2>
<p>There have been a number of changes in the Python packaging/distribution world in the past few months. I&#8217;m not involved in any of the relevant projects, but since my installation guide is quite popular, I get emails from some folks every so often that are. One of the biggest changes is the reintegration of the Distribute fork of setuptools back into the main project. This also means that setuptools &#8211; the main project &#8211; is getting a lot more love, which means more&nbsp;updates.</p>
<p>The installation instructions for the Microsoft C++ Compiler for Python 2.7 package says that it requires <strong>setuptools 6.0 or later.</strong> I had a crusty old version from who knows when. Updating pip and setuptools is a little weird, but it&#8217;s not that difficult. I actually wrote a <a href="http://engineer.readthedocs.org/en/master/upgrade.html">separate guide for that</a> as part of the Engineer 0.5.0&nbsp;documentation.</p>
<p>There are more details there, but it basically boils down to executing two commands: <code>python -m pip install -U pip</code> followed by <code>pip install -U setuptools</code>. When you do that, you should see some output like&nbsp;this:</p>
<div class="codehilite"><pre>C:\Users\Tyler\Code&gt; python -m pip install -U pip
Downloading/unpacking pip from https://pypi.python.org/packages/py2.py3/p/pip/pip-6.0.2-py2.py3-none-any.whl#md5=26404d27a64a40d4c358a2405b16d043
Installing collected packages: pip
Found existing installation: pip 1.5.2
Uninstalling pip:
Successfully uninstalled pip
Successfully installed pip
Cleaning up...
C:\Users\Tyler\Code&gt; pip install -U setuptools
Collecting setuptools from https://pypi.python.org/packages/3.4/s/setuptools/setuptools-8.2.1-py2.py3-none-any.whl#md5=a0582adbe0c56b3945570049b8d7c953
Downloading setuptools-8.2.1-py2.py3-none-any.whl (551kB)
100% |################################| 552kB 975kB/s ta 0:00:01
Installing collected packages: setuptools
Found existing installation: setuptools 2.2
Uninstalling setuptools-2.2:
Successfully uninstalled setuptools-2.2
Successfully installed setuptools-8.2.1
</pre></div>
<p>Congratulations, your pip and setuptools installations are now upgraded. As I note in the <a href="http://engineer.readthedocs.org/en/master/upgrade.html">Engineer 0.5.0 upgrade guide</a>, &#8220;if you’re using virtualenv, you may need to upgrade pip and setuptools in your virtualenv as well as the &#8216;global&#8217; (outside the virtualenv) versions.&#8221; You should be able to avoid doing this for all new virtualenvs by upgrading virtualenv itself (<code>pip install -U virtualenv</code> &#8211; version 12.7.8 is the latest as of this writing). Once it was upgraded my new virtualenvs got the correct updated versions of pip and setuptools. If you don&#8217;t want to recreate your virtualenvs, then you can just upgrade the ones you&nbsp;need.</p>
<p>Once pip and setuptools are upgraded, try installing the previously failed package again. You should see a bunch of output like&nbsp;this:</p>
<div class="codehilite"><pre>C:\Users\Tyler\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\Bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\Libs /LIBPATH:C:\Users\Tyler\.virtualenvs\test\libs /LIBPATH:C:\Users\Tyler\.virtualenvs\test\PCbuild /EXPORT:init_diff_tree build\temp.win32-2.7\Release\dulwich/_diff_tree.obj /OUT:build\lib.win32-2.7\dulwich\_diff_tree.pyd /IMPLIB:build\temp.win32-2.7\Release\dulwich\_diff_tree.lib /MANIFESTFILE:build\temp.win32-2.7\Release\dulwich\_diff_tree.pyd.manifest
Creating library build\temp.win32-2.7\Release\dulwich\_diff_tree.lib and object build\temp.win32-2.7\Release\dulwich\_diff_tree.exp
Successfully installed dulwich-0.9.8
</pre></div>
<p>Congratulations, you can now install source Python packages that include C extensions (like <a href="https://www.samba.org/~jelmer/dulwich/">Dulwich</a>)!</p>
<h2>The Future: Binary&nbsp;Wheels</h2>
<p>Now, the fact that you need to install some separate dependency on Windows in order to install some Python packages clearly sucks. Fortunately, there are ways that package distributors can remove this need. There is a new package format, called <a href="https://pypa.io/en/latest/peps/#pep427s">Wheel</a>, which includes pre-compiled versions of C extensions. If the Dulwich package maintainer produced a Wheel in addition to the source distribution, then users wouldn&#8217;t all need to install the Microsoft C++ Compiler for Python 2.7.<sup id="fnref:4"><a class="footnote-ref" href="#fn:4" rel="footnote">4</a></sup> Wheels can be installed using pip version&nbsp;1.4+.</p>
<p>If you release packages on PyPI, consider creating a Wheel if your package has a C extension. There&#8217;s a great guide for distributing your Python projects, including creating Wheels, in the <a href="http://python-packaging-user-guide.readthedocs.org/en/latest/distributing.html">Python Packaging User Guide</a>.</p>
<h2>Addendum:&nbsp;dulwich-windows</h2>
<p>In the event you&#8217;re using Windows and need <a href="https://www.samba.org/~jelmer/dulwich/">Dulwich</a>, but don&#8217;t want to fool with following the steps above, I <a href="dulwich fork">forked the repository</a> and published a Wheel (my first!) &#8211; it&#8217;s <a href="https://pypi.python.org/pypi/dulwich-windows">dulwich-windows</a> on PyPI. The <em>only</em> change to the code is a few minor changes to the setup file to differentiate it from the official Dulwich package. You can see those changes on the windows_wheel branch in <a href="dulwich fork">my fork</a>. Feel free to install it (<code>pip install dulwich-windows</code>). I may or may not keep it up to date, though, so I recommend using the official Dulwich releases if&nbsp;possible.</p>
<p><em>This guide was last updated December 31, 2014.
It was tested on a Windows 8.1 machine. If you notice any errors or missing/out-of-date information,
please let me know at: __tyler <span class="caps">AT</span> tylerbutler <span class="caps">DOT</span>&nbsp;com__.</em></p>
<div class="footnote">
<hr />
<ol>
<li id="fn:1">
<p>For the record, I don&#8217;t recommend following most of the answers on Stack Overflow at this point. Many are quite old, and as you&#8217;ll see when you read on, there&#8217;s a much simpler solution.&#160;<a class="footnote-backref" href="#fnref:1" rev="footnote" title="Jump back to footnote 1 in the text">&#8617;</a></p>
</li>
<li id="fn:2">
<p>Well, since it&#8217;s Microsoft, it was likely a whole team of people!&#160;<a class="footnote-backref" href="#fnref:2" rev="footnote" title="Jump back to footnote 2 in the text">&#8617;</a></p>
</li>
<li id="fn:3">
<p>If you happen to know who at Microsoft was responsible for this, please let me know, because I want to buy them something nice.&#160;<a class="footnote-backref" href="#fnref:3" rev="footnote" title="Jump back to footnote 3 in the text">&#8617;</a></p>
</li>
<li id="fn:4">
<p>I do think that the Wheel would need to be built on a Windows box with the compiler installed, though I am not sure about that. If I&#8217;m right, this would certainly be a blocker since many package maintainers don&#8217;t have easy access to a Windows box.&#160;<a class="footnote-backref" href="#fnref:4" rev="footnote" title="Jump back to footnote 4 in the text">&#8617;</a></p>
</li>
</ol>
</div>
Wed, 31 Dec 2014 00:00:00 -0000http://www.tylerbutler.com/2014/12/installing-binary-python-packages-on-windows/Updated Python Installation Guidehttp://www.tylerbutler.com/2014/10/updated-python-installation-guide/ <p>I just made a major update to my <a href="/2012/05/how-to-install-python-pip-and-virtualenv-on-windows-with-powershell/">Python Windows installation guide</a>, which remains my most popular post. Things have gotten a lot simpler over the past few months since the distribute fork of setuptools was integrated&nbsp;back.</p>
Tue, 07 Oct 2014 00:00:00 -0000http://www.tylerbutler.com/2014/10/updated-python-installation-guide/Stack Traces →http://rustyshelf.org/2014/08/07/thoughts-on-swift-from-an-idiot/ <blockquote>
<p>Yes I know, ha ha Null Pointer, Java, <span class="caps">LOL</span>. But that’s an exact line number friends. What did the user do? They tapped the subscribe button. Which page where they on? The Podcast Dialog. Zero ambiguity. Guess how many of our Android crashes we get that for? 100%. In iOS we’d be lucky if even 30% of our crashes had stack traces we can line up to actual things we can then reproduce. So most iOS crashes today involve me becoming House <span class="caps">MD</span> and poking the code for hours, only to figure out that like always, it’s never&nbsp;Lupus.</p>
</blockquote>
<p>It astounds me that iOS debugging is so&#8230; <em>medieval&#8230;</em> That said, JavaScript stack traces can be just as bad, depending on the browser. I think good stack traces are a <em>requirement</em> for actually shipping software. Here&#8217;s hoping iOS, and browsers, catch up&nbsp;soon.</p>
<div>
via <a href="http://inessential.com/2014/08/27/a_rant_about_stack_traces">Brent Simmons</a>
</div>
Tue, 07 Oct 2014 00:00:00 -0000http://www.tylerbutler.com/2014/10/stack-traces/