I haven’t blogged in over a year. I’ve been busy working at Teladoc.com, which I joined last March, and parenting.

I feel healthier because I’ve run 4-5 days a week since May (though I’ve been slacking slightly the past few weeks since the belated arrival of winter weather disrupted my after-work run ritual). I highly recommend regular exercise to everyone to boost physical and mental health. The best way to exercise consistently is to make it a regular habit. Pick a few specific times each week and commit to regularly exercising then. Something is better than nothing, so just go out and sweat, but don’t sweat it if you’re not up for a full workout each day. Just work out consistently.

If you’re a programmer, you might like my ever-growing list of tech resources. I’m eagerly learning Elm and Elixir/Erlang, but — because I work long hours and spend much of my weekends with the kids — it has been hard to devote as much time to them as I would like. I’m excited about functional programming; it’s the future because functional code is much more maintainable and scalable than object-oriented code.

A Hedgeye colleague who has suffered similar pain from Chef as I have (especially unstable APIs) suggested we switch to Docker, and we’re now pursuing that. So far, we both like it, though we haven’t yet done anything complicated.

To help our future selves and our colleagues use Docker, I’m going to document some stuff here.

Docker will then pull down a Ubuntu image (unless it has already done so, in which case it will use what you already have), spin up an instance running just bash console (the command you told it to run), and drop you into the console session.

When you’re finished interacting with your instance, type exit and it will log you out and then terminate the instance. It terminates because Docker’s approach is to spin up a container running just one process and shut that container down when the initial process is no longer active. From the initial process, you can, of course, spawn additional processes. But whenever the special initial process closes, your container dies.

Lately, I’ve been doing a lot of Python data analysis. Though Python has many strengths, package management has long been a nightmare.

Continuum Analytics has created several wonderful free open-source projects, perhaps most notably Anaconda, which makes installing Python and packages much, much easier, esp. if you want to maintain multiple Python environments, which you probably do, esp. if you want to run both Python 2 and Python 3.

I’ve just hit on a workflow that enables me to keep my environment up to date without risk of breaking stuff.

I currently have two environments, the default 2.7 environment and a 3.4 environment I use most of the time:

The update succeeded, so my environment is now totally up to date. Thanks, Continuum Analytics! But the update could have failed. Or it could have succeeded but one or more of the updated packages could have broken my applications in ways I don’t like, causing me to want to roll back to where I began and update more selectively.

Having a snapshot of my environment and the ability to instantly recreate it gives me peace of mind.

At home, I run Linux machines, my wife is on a Mac, and my kids and in-laws are on Windows laptops. (I’ll transition my kids to Linux as they move into programming.)

Because of this heterogeneity, I like to format my external hard drives with multiple partitions, each for a different OS.

But this can be a huge pain. I formatted a 3 TB hard drive with a Windows partition, a Linux partition, and space for a Mac (HFS+) partition. But my wife’s MacBook Pro’s Disk Utility refused to create an HFS+ partition on the third physical partition, complaining that the hard drive has a Master Boot Record. I wasn’t trying to create a bootable partition, but Mac OS didn’t care. Using my Linux machine (and the “hfsprogs” package), I managed to format the partition as HFS+. It shocked me that Linux could create a Mac-formatted partition where a Mac couldn’t.

My wife’s MacBook agreed it was an HFS+ partition in good state, but it still refused to let TimeMachine back up to it because it was a non-journaled HFS+ partition. GParted can’t create a journaled HFS+ partition.

I finally surrendered and threw away all my partitions and let the MacBook Pro take the first physical spot on the hard drive. TimeMachine is finally running. I won’t know whether I can use the unformatted 2TB of space for Windows or Linux till it finishes. Proprietary OSes are so annoying!

Had fun today at work tuning a Postgres database that has gotten very slow over the years as it has accumulated many gigabytes of data. This app is frequently rendered inoperable by just one or two users visiting its home page, which is obviously a bad situation. (Luckily, it’s an in-house tool used almost exclusively by a single user, which is why it hasn’t received more love before now.)

Over the past week, I’ve been recording the slowest queries, and today I started attacking them. The easiest-to-fix were the ones caused by missing indexes. Another problem I found was unnecessary overhead from two compound indexes that were indexing the same two columns with opposite orderings; I turned one into a single-column index, which should produce similar read performance and superior write performance.

A third fix I proposed was adding a field for the calculated value of md5(email). Some queries have been doing full-table searches of md5(email). I don’t understand why that’s necessary, but having to calculate md5() for every row in the table and then scanning the whole table sounds pretty inefficient. So I created a named function for calculating md5(email) and a trigger that calls the function whenever a table record is added or modified. Doing this at the database layer makes sense because Rails doesn’t need to know anything about md5(email).

I also created my first Postgres materialized view today. Another query can occasionally take 40+ seconds on our server. The same query normally runs orders of magnitude faster, so I’m not sure what causes such long delays. But it’s doing a join that involves calculating a count on a large table. My first thought was to add a counter cache, but that didn’t make sense when I looked at the table layout. I instead made a materialized view, which worked well on my static copy of the production database. But when I went to the Postgres documentation, I discovered two flaws with Postgres 9.3’s materialized view implementation: 1) Updating the materialized view is a manual process; and, 2) Updating the materialized view takes a full lock on the view. So I’m not sure it’s worth pushing to production, but I’m glad to read that Postgres devs are already working to improve the implementation of materialized views.

But I can’t figure out how to replace a multi-line code block in a file. The articles I’ve read suggest Chef tries to force users into replacing whole files. cassianoleal answers a question about how to do so with, “As you said yourself, the recommended Chef pattern is to manage the whole file.” Why must we copy entire files to replace one block of code with another? Chef 11 apparently includes “partials,” which let you insert multi-line code elements. Chef::Util::FileEdit also lets you do that. But the ability to insert multiple lines doesn’t enable deleting multi-line code blocks. I probably could copy the entire file into memory, search-and-replace the multi-line segment with a regex and write the modified file back, but shouldn’t this be a built-in Chef tool?

Today at work, I decided to roll back the previous few commits I had made, but I didn’t want to git reset --hard and throw away history or mess up anyone else who might have pulled from my branch, so I decided to git revert, but I wasn’t quite sure the syntax.

I pulled out my normally reliable name-brand search engine and, after an unusually long search, found the “answer.” But it wasn’t quite right. It failed to revert one of the commits I wanted to revert. So I’m putting the answer here in hopes it saves someone else some pain.

For my OS, pkg_php_mbstring is nil. But that means “package pkg_php_mbstring” translates to “package nil,” which is illegal.

Fixing this tiny bug would be easy if I controlled the cookbook, but Librarian does. I cloned the Github repo and modified the offending lines, but Librarian wouldn’t let me use my version of the cookbook because it’s included by another cookbook, which is included by another cookbook. I would have to clone them all and redefine the entire dependency chain to get my tiny bug fix in.

one of the major annoyances with Chef: since librarian-chef managed these cookbooks and the directories
in which they lived, the workflow for editing them was tedious and hackish. Editing stuff you don't own
[involves]:
* clone down other cookbook
* move librarian-managed cookbook somewhere else
* symlink your cloned cookbook in its place so knife can find it
* knife cookbook upload <cookbook>
* whack moles
* git commit
* remove the symlink to the cloned cookbook
* put the librarian-managed cookbook back where it was (or delete it)
* bundle exec librarian-chef update <- to update your Cheffile.lock to have the right version of the
cookbook you just edited
* bundle exec librarian-chef install <- to install the version you just specified
There are ways to make the above process shorter such as scripting steps 2, 3, 7, and 8 as well as
saving steps 9 and 10 until you are totally done working. And, to be fair, librarian-chef served the
very important purpose at one time. However, the process of editing stuff you don't own is still
tedious and less than ideal.

I really want to love Chef, but I keep hitting issues like this. Sigh.

Many, many years ago, I created a Slicehost server on which I host this blog post, many of my other websites and my mailserver. I used to keep it up to date. But these past four years, I’ve been remiss about maintaining it, initially due to laziness and later due to the immensity of the challenge of upgrading out-of-date software on a cloud server when the disk image and OS/packages must be kept in sync, compounded by how hard I’ve worked on my day jobs these past few years. This challenge was further complicated by Ubuntu ending upgrade support for the version I’m running and by Slicehost getting bought by Rackspace. Slicehost used to provide detailed instructions for simultaneously updating one’s Ubuntu version and one’s Slicehost disk image. That no longer exists.

So I’ve decided I should build a new server and migrate my data and services one-by-one to the new server.

Building a new server sounds like a job for Vagrant and Chef, and I found a vagrant-rackspace Vagrant plugin and a chef-mailserver cookbook for building a mail server. I’ve decided to give these a try. I’ll record my efforts here in case they’re helpful to anyone.

I like the choices made for chef-mailserver (postfix, dovecot, amavisd, clamav, spamassassin, and postfixadmin because they’re almost identical to the decisions I made years ago for my mail server. So these are time-tested technologies, and I’m familiar with most of them. Unfortunately, my existing database is MySQL while chef-mailserver uses PostgreSQL. But, while that’s a migration headache, I prefer Postgres, so I can live with that.

On my Debian desktop, I discovered the latest version of Vagrant is available as a package, so I installed it. I then ran vagrant plugin install vagrant-rackspace and got a “Failed to build gem native extension” error. The problem was that I didn’t have the ruby1.9.1-dev package installed. (I use Rbenv and generally don’t like to install/use Ruby packages from package manager.)

With vagrant-rackspace (0.1.6) installed, I followed the instructions at https://github.com/mitchellh/vagrant-rackspace, first running vagrant box add dummy https://github.com/mitchellh/vagrant-rackspace/raw/master/dummy.box, then creating a Vagrant file, for which I had to find my API key in my Rackspace Cloud account. I also chose some non-default settings, like setting rackspace_region to :ord because that’s where my existing server is, which could make migrating data/apps simpler and cheaper. (I’m not sure whether transferring data across regions counts as free private bandwidth or non-free public bandwidth.)

…After long break for work, kids, and a basement flooded by a dead hot water heater…

I used Vagrant to create a Digital Ocean “droplet.” I first created an account at DigitalOcean.com and provided my credit card information. I then uploaded my desktop’s public SSH key to the “SSH Keys” tab and got my Client ID and API Key from the “API” tab.

I then filled in the Vagrantfile and overrode some defaults. Normally, I install the latest Ubuntu LTS release, but since a new LTS release is right around the corner (14.04 LTS is scheduled for April 2014), I decided to start with the current hotness (13.10) and upgrade to the LTS soon.

I created my “droplet” with vagrant up --provider=digital_ocean. I saw the information Vagrant stored about my new machine in .vagrant/machines/default/digital_ocean.

Vagrant no longer provisions machines with Chef by default. I forgot to do so, so after installing the vagrant-omnibus plugin (vagrant plugin install vagrant-omnibus), I added config.omnibus.chef_version = :latest to my Vagrantfile and ran vagrant reload --provision. It installed Chef 11.10.4-1.ubuntu.13.04.

I now want to use Chef to provision the server. There are multiple options for doing so. I’ve previously used Berkshelf and initially decided to do so again here. I ran gem install berkshelf and created a Berksfile like the following:

site :opscode
cookbook 'build-essential'

before realizing Berkshelf 2 (which is what gem install berkshelf installs) is now deprecated in favor of Berkshelf 3. So I ran gem uninstall berkshelf and followed the Berkshelf 3 instructions. I also had to add vagrant-digitalocean to both the Gemfile and Vagrantfile and change site :opscode to source "https://api.berkshelf.com". After doing so, I ran bundle exec vagrant reload --provision and got the expected output (on my fourth attempt).

But I gave up on Berkshelf for this project because its documentation currently warns against using version 2 but doesn’t document version 3. This situation should improve. But I’ve decided to give up the comfort of Berkshelf because its API appears in flux.

Installing cookbooks with knife is wonderful because knife usually identifies dependencies and installs them too. Also, knife will auto-commit your cookbooks into Git for you.

Before the above commands worked, I hit a snag (ERROR: IOError: Cannot open or read /home/jimmy/Git/jimmy-chef-mailserver/cookbooks/build-essential/metadata.rb! and ERROR: Errno::ENOENT: No such file or directory - /home/jimmy/Git/jimmy-chef-mailserver/cookbooks) that I fixed by force-deleting branches with git br -D chef-vendor-build-essential and git br -D chef-vendor-mailserver per the solution recommended here.

When I added the cookbooks' recipes to my runlist in Vagrant and re-ran vagrant reload --provision, this line in one of the recipes choked:

hosts = search(:node, "*:*").map { |n| n["ipaddress"] }

because the recipe was written for Chef Server only, not Chef Solo, which I knew because it complained:

Chef::Exceptions::PrivateKeyMissing
-----------------------------------
I cannot read /etc/chef/client.pem, which you told me to use to sign requests!

Googling let me to chef-solo-search, which I installed with knife cookbook github install edelight/chef-solo-search, but then I started hitting problems because my setup was trying to look for a Chef Server.

I think I’ll stop using Chef-Solo and Berkshelf through Vagrant and instead use Vagrant just to boot up a minimal machine and then handle provisioning outside Vagrant. The article above recommends Test Kitchen: “There exists a better solution for provisioning virtual machines with Chef – Test Kitchen. Test Kitchen still leverages the amazing resolving power of Berkshelf under the covers, so switching is easier than ever. By switching to Test Kitchen, you are also no longer tied to Vagrant. You can use Berkshelf with your favorite cloud provider or bare metal!”

I’ve been building Linux machines by hand for over a decade and have always found the process time-consuming, error-prone, and hard-to-replicate. The new hotness in “DevOps” is “infrastructure as code,” i.e., writing code that creates, provisions, updates, and manages our machines (servers/laptops/desktops).

When the logic for building machines lives in code, rather than solely in developers' brains, that logic is obviously easier to modify and reuse. In “the old days,” for example, replication meant disk images. Copying hard drives bit-for-bit worked well for copying one machine’s software to an identical piece of hardware. This was useful for large companies that issued all employees identical computers but useless for upgrading to a more powerful — or just different — machine. Anything other than putting identical software on identical hardware required redoing everything by hand.

Infrastructure code now lets you re-run the steps used to build a machine, and those steps (possibly with tweaks that work around any hardware or software differences between installations) will rebuild the new machine similarly to the old machine, despite underlying hardware differences. Even better, shared public cookbooks (i.e., collections of formulas/recipes for installing and configuring various pieces of software on various operating systems) enable everyone to leverage the work of others, saving huge amounts of time formerly wasted trying to configure software by trial-and-error or Googling for answers.

Two popular tools for building “infrastructure as code” are Chef and Puppet. Though each has its advantages, I chose Chef, partly because it has a strong community and partly because another developer at my firm used it on an earlier project. A third tool, BOSH, looks promising because it actually attempts to do more than Chef/Puppet (which are mainly for building and maintaining individual machines), as this diagram shows; but BOSH currently feels less mature than Chef. (A good talk on BOSH.)

I’ve recently built my first Chef-scripted server, and I’m very happy with the results. I was tasked at work with replicating an existing server (the one we test our code on using Jenkins) so we could get it up and running quickly if it ever fails again (as it did a few months ago). This suggests another huge advantage of “infrastructure as code”: Minimal downtime. When a hand-crafted server goes down, it can take days or weeks to get a new machine back up with full functionality. In fact, unless it has been recently and fully backed up, it can prove impossible to recreate the machine. When a Chef-scripted server goes down, you can re-run the script and build a new machine in minutes or hours, not days or weeks.

I was able to script the entire process of creating and provisioning the new machine (on both a physical box we owned and on a new AWS virtual machine), including the setup of PostgreSQL, MySQL, users, shadow passwords, custom user RVM and Ruby installs, etc. Though I can’t show you the code, I’d like to share the big picture on how to do the same.

Chef (site): The Ruby-based DSL (domain-specific language) for provisioning machines via scripts. I found the “Resources and Providers Reference” (http://docs.opscode.com/chef/resources.html) so helpful that I printed it out, despite it being book-length. There’s also good documentation of the “Recipe DSL” (http://docs.opscode.com/chef/dsl_recipe.html). And I found a cookbook for most everything I wanted to install/configure on my machine. A good place to start looking for cookbooks is https://github.com/opscode-cookbooks, but there are many cookbooks not created by Opscode (just Google “xyz cookbook,” where “xyz” is MySQL, Jenkins, PostgreSQL, or whatever else you might want). Learn Chef offers tutorial videos. And the free book “Getting started with Chef” looks helpful.

Chef Solo (site). A stripped-down Chef tool that lets you provision machines using Chef scripts but without the heavy overhead and multiple machines involved in a full Chef environment. Self-description: “an open source version of the chef-client that allows using cookbooks with nodes without requiring access to a server. chef-solo runs locally and requires that a cookbook (and any of its dependencies) be on the same physical disk as the node. chef-solo is a limited-functionality version of the chef-client.” Useful resources: Railscast on Chef Solo Basics.

VirtualBox (site): A tool that lets you create and run virtual machines on your desktop or laptop. Not intended for production usage but very convenient while creating Chef scripts that will eventually run elsewhere. Also used by development teams to create development sandboxes that can run identically on various machines with different operating systems and hardware/software configurations.

Packer (site): Though I didn’t use Packer, I’m mentioning it here because it looks useful. It “lets you build Virtual Machine Images for different providers from one json file. You can use the same file and commands to build an image on AWS, Digital Ocean or for virtualbox and vagrant” (“Building Vagrant Machines With Packer”).

These three blog posts are useful for seeing how Chef-Solo, Vagrant, and Berkshelf work together: 1, 2, and 3.

I picked a Vagrant “box” that matched what we’re running on our current Jenkins server (i.e., a vanilla 64-bit install of Ubuntu 12.04, which we picked because it’s a LTS or “long-term support” release, meaning it will continue receiving security updates for several years).

Using Berkshelf, I installed some cookbooks, like chef-rvm, and configured them in my Vagrantfile.

I then created my own cookbook with two recipes, one that runs early in the provisioning process to create users, etc. and one that runs near the end of the provisioning process to install and configure additional packages, like ack-grep, emacs (because some developers have poor taste), imagemagick, ghostscript, Node.js, libmagick, and libssl. See Chef’s recipe DSL documentation and this guide to authoring cookbooks for the many things you can do in your recipes. You can set shadow passwords, set environment variables, run arbitrary Bash or Ruby code, install Ruby gems, etc.

There are even methods available in Chef::Util::FileEdit for manipulating files in various ways, like search-and-replace. Though it’s better to configure your machine’s apps via chef.json settings passed by the Vagrantfile to your Chef cookbooks, you can use Chef::Util::FileEdit to directly modify textual configuration files. I found I had to do so to enable password-based SSH logins.

I also installed the foodcritic Ruby gem, a linting tool for Chef cookbooks, which I found slightly useful on my small project; I suspect it could be quite helpful on a larger project.

I hit a few issues I want to flag (esp. my colleagues and future me):

Installing RVM: Installing RVM and multiple Ruby versions on different users' accounts was hard. I tried using chef-rvm but couldn’t get it running quickly with Chef-solo. That could have been, in part, because I was still pretty new to Chef. Whatever the reason, I found the answer in this blog post from CloudSpace.com. The key was adding 'vagrant' => { 'system_chef_solo' => '/usr/local/ruby/bin/chef-solo' } to chef.json in my Vagrantfile.

Installing AWS CLI tools: I struggled to install the two sets of AWS CLI tools I needed till I discovered aws-cli. Then it was as trivial as running pip install awscli. (I happened to already have pip installed; if you don’t, you can install pip.)

AWS security groups: Amazon (AWS) uses “security groups” to manage privileges. You probably will need to explicitly open up various ports (80, 8080, 443, etc.) on your AWS machine via AWS’s web console. You can create multiple “security groups” in the AWS console, assign privileges to security groups, and then (in Vagrant) assign various security groups to each AWS VM you create. And, as vagrant-aws says, “If you have issues with SSH connecting, make sure that the instances are being launched with a security group that allows SSH access.” We also needed to set that in the AWS console before I could SSH into my new virtual machine.

AWS elastic IP addresses: Amazon (AWS) uses “elastic IP addresses.” What this means is that Amazon provides your box with TWO IP addresses. The IP address your box believes is its public IP address is actually a private IP address inside Amazon’s network. To access your box from the Internet, you must use the other IP address Amazon provides, which Amazon maps to your private internal IP address. Your box won’t know anything about its true public IP address, but you’ll need to use that to SSH into it (outside of Vagrant) or hit any of its ports for viewing websites, etc. I figured out the actual public IP from the “Connection to xx.xxx.xxx.xx closed.” message I saw every time I exited my “vagrant ssh aws” session.

AWS region in Vagrantfile: The AWS region specified in your Vagrantfile should NOT include the letter suffix. For example, aws.region="us-west-2a" should be aws.region="us-west-2" See this discussion.

Verbose Vagrant output: To see more verbose Vagrant output, prefix your vagrant commands with VAGRANT_LOG=debug. I found this incredibly useful when I tried to deploy to AWS and the process kept hanging at “[aws] Waiting for SSH to become available…” After adding the environment variable, I suddenly saw rich feedback which was invaluable for diagnosing and fixing the underlying problems.

Enabling passwordless sudo: I found it useful to enable passwordless sudo. I did so using the sudo cookbook.

One Chef script, multiple Vagrant providers: Running one set of scripts across multiple Vagrant providers. Initially, I built my scripts on a VirtualBox virtual machine running on my laptop controlled by Vagrant on my laptop. When I was ready to deploy to a new virtual machine on AWS, most of the script remained the same, but certain things needed to change. To change/override configuration settings for a specific provider, you define a config.vm.provider inside your Vagrantfile. For example, I used config.vm.provider :aws do |aws, override| to specify settings that applied only to my AWS machine. You can also define config.vm.define to name your various instances. I had a config.vm.define "aws" and a config.vm.define "local" and could call vagrant using those names (e.g., vagrant ssh aws or vagrant provision local). I found this page on DRYing up Vagrant files helpful for understanding how to use the same script on multiple machines running on different providers.

Broken VirtualBox required re-installation: My VirtualBox installation stopped working, probably after Mac OS upgrades of some kind. I opened the VirtualBox GUI and found the error message “NS_ERROR_FAILURE (0x80004005).” Further investigation revealed that VirtualBox also gave me the error “kernel driver not installed (rc=-1908).” I Googled these and learned that VirtualBox is highly prone to getting corrupt. There were tons of Google hits for these errors. I uninstalled then reinstalled VirtualBox from the same VirtualBox.pkg I had previously installed it with. After doing so, vagrant up and vagrant ssh both worked! Moral of the story: Updates to Mac OS can break VirtualBox, so you may need to delete/reinstall VB.

Flawed Chef scripts that appear to succeed: As you’re becoming familiar with Chef and adding to your scripts, you’ll probably find your scripts failing occasionally. Beware “fixing” it by rearranging the order of your script because you may well find that your script now runs to completion but later discover that it didn’t run everything you believed it ran. When you run vagrant provision, it skips resources it considers already provisioned. I recommend occasionally wiping out (vagrant destroy) your virtual machine and re-building your machine from scratch (vagrant up) to guard against false positives.

Use apt cookbook but don’t put ‘sudo apt-get upgrade’ in your Chef script: If you’re running Ubuntu on your VM, you’ll want to install the apt cookbook to keep your apt cache up to date. But I still run sudo apt-get update and sudo apt-get upgrade manually because this advice recommends against putting “apt-get upgrade” in a Chef script.

Thanks to my employer, Hedgeye, for giving me time to write this blog post and to my wonderful colleague Scott Smith for helpful comments that make this less painful to read. All remaining errors are completely Scott’s fault, of course.

You also can’t just call the function because it’s a method of the john object, not a global function:

getName(); // ReferenceError: getName is not defined

Since john and mary are both people, it would make sense for .getName() to be defined on their common prototype, but let’s ignore that solution for now and focus on how to create a function that lets us reuse john.getName to return the name of the mary object.

The simplest way is probably with .call(), which calls the function (in this case, john.getName) with a context of whatever parameter is passed into .call(), in this case, mary:

john.getName.call(mary); // returns "Mary"

Another way is with .bind(). We create an entirely new function by grabbing the function we want to use (john.getName) and then binding it to the object on which we want the function to be called (mary):

var getMaryName = john.getName.bind(mary);

Now, when we call/invoke the getMaryName function, it will call john.getName and set “this” (in “function() { return this.name; }”) to the mary object, so the correct value gets returned:

getMaryName(); // 'Mary'

This is an entirely new function, not a method of the mary object. So, mary still does not know how to return her own name:

The function belongs to the global scope and is disembodied from the john and mary objects. And we have cluttered up the global namespace with another variable name and consumed additional memory by creating a new function you may never use again. A better approach is probably to create the new function and immediately call it. Let’s delete the variable and function:

getMaryName = null;

And instead define and execute the new function in a single step:

(john.getName.bind(mary))(); // returns "Mary"

This is a roundabout way of doing what john.getName.call(mary) does. But there will be times when you actually do want to create a new function that is bound to a particular object.

A more verbose, but possibly useful, way to do this is by defining and immediately executing an anonymous function into which you pass the object you wish the function to bind to, in this case, mary:

To summarize: func.bind(obj); takes a function (func) and an object (obj) and returns a new function that, when called, calls func with “this” set to obj.

You can also use .bind() to insert additional arguments at the beginning of a method call’s arguments list. Let’s give John the ability to set and get an array of his favorite people in the world and make sure — because everyone loves their mother and would never want to forget their mom — the array always holds at least John’s mom. We’ll create a method that takes the arguments array, adds ‘mom’ at the beginning, and then stores the array in john.favorite_people:

This obviously outputs “Hello, world!” to console.log. But this is completely deterministic, procedural code, and that’s not always the case in Javascript.

If you’re making an AJAX call from a client/browser to a server or pulling a value out of an IndexedDB database, you must process the results via a callback that fires after an unknowable period of time passes. To simulate this behavior, let’s change storeHello():

This adds an artificial half-second delay before what we really want storeHello() to do happens. What now happens?

You might suspect console.log now spits out “world!Hello, ” but it actually spits out just “world!” Why? Because console.log() fires before the setTimeout() concludes and calls its callback. So the string is printed out without ‘Hello, ’. hello_world_string eventually gets set to “world!Hello, ” but only after it has already printed. Code execution doesn’t stop half a second when storeHello() is called and its setTimeout() sets a half-second delay. Code execution continues immediately with the next command, storeWorld(), and then the next, console.log().

If you really need the string components to print in the correct order after all functions have finished processing, you need to do additional work. One way to guarantee the correct order is by calling storeWorld() inside storeHello()’s callback function and calling console.log() inside storeWorld():

An alternative to placing callbacks inside callbacks would be to create an object responsible for gathering the string components, joining them in the proper order, and doing something with the full string (e.g., logging it to the console) after it received the final component. This approach would likely be several times faster because components can be gathered simultaneously (in parallel), rather than sequentially (in series) as happens when callbacks invoke callbacks.

Our next stop on our trek through Javascript array iteration methods (begun in Part I) is Array#filter. We’ll start by creating a function that takes a value and returns true or false, depending on whether the value is or is not greater than 2:

var greaterThan2 = function (val) { return val > 2; };

We can now filter any array to find all values > 2 by passing greaterThan2 as a callback to Array#filter:

var arr = [0,1,2,3,4];
var arr2 = arr.filter(greaterThan2);

arr2 is now [3,4]. arr is unchanged.

Like most of Javascript Array’s iteration methods, Array#filter passes the array element’s index position as the second argument when it invokes the callback. So you can filter not only on the element’s value but also on its position within the array. To see this, we’ll create a callback that returns true if and only if the element’s array index is < 3 (i.e., 0, 1 or 2):

And that completes our tour of JavaScript array’s iteration methods. The one method I did not cover is Array#reduceRight. It’s basically Array#reduce, except that it processes the array starting at the end and working back to the beginning, rather than the other way around. This is pretty unimportant because you could always just call Array#reverse before calling Array#reduce.

But what if what you really want to do is test whether every atomic value in the three subarrays held by abc is less than 5? You could (incorrectly) try this:

abc.every(lessThan5); // false, but not testing what you believe

To see that it’s not working as we want it to, let’s create this callback:

var greaterThan0 = function (val) { return val > 0; };

Since every value in every array held by abc is > 0, we want this callback to return true, but it still returns false:

abc.every(greaterThan0); // false

abc.every is looping over an array of arrays, not an array of primitives (integers). What we really want is to have the callback called once for each array in abc call array.every(greaterThan0) on whatever array it receives. We need an intermediate callback, or a callback that calls .every() with another callback. How might we do this?

What if we don’t know till runtime how many levels down into subarrays we must descend before we hit primitive values? Then the callback can’t know in advance whether it’s being handed an array of arrays or an array of primitives. But it can test what it has been handed and either call itself (if it has received an array of arrays) or call greaterThan0 (if handed an array of primitives):

We’ve assumed the original array is at least an array of arrays. If it’s a simple array, the callback fails. How can we correct this? Instead of assuming we received an array and testing what its child is, we can test what we’re actually handed and respond accordingly:

This returns the value 10 in two lines, versus the previous four lines. Array#reduce takes a callback and a starting value (here 0). If we had wanted to multiply the numbers 1 through 5 together, we could have done the following:

This returns 120 (1*2*3*4*5). arr passes in 0, then 1, then 2, etc. up to 4. Each iteration, this becomes newVal. We add one to it, then multiply it by the return value of the last iteration. So it goes: 1, 2, 6, 24, 120.

Remember our old friend?

var plusOne = function (val) { return val + 1; };

Well, we could use this to compose the solution above, rather than hardcode newVal + 1:

I’m pleased to announce JimmyJukebox (Ruby gem; Github code) now offers classical music as well as jazz. Other improvements include the ability to limit the number of songs downloaded, the ability to erase a song you don’t like, and the ability to replay the previous song.

When scientists are baffled by facts that contradict their strongest theories, science is likely on the cusp of new insights.

That’s why I was saddened when scientists discovered the Higgs boson — or “god particle.” It meant CERN’s $20 quadrillion (or whatever) Large Hadron Collider did not uncover new facts that would force physicists to re-think their theories and provide clues how they might bend those theories closer to reality. It merely provided greater support for what physicists already believed.

This process of gaining knowledge from proof of our ignorance is also why it’s fabulous that “The newly discovered LQC is so enormous… that theory predicts it shouldn’t exist.” The largest structure known to man consists of “73 quasars and spans about 1.6 billion light-years in most directions, though it is 4 billion light-years across at its widest point.” Even better than how mind-blowingly amazing that is, we have absolutely no clue how it got so massive because our best theories say it can’t exist! So we’ve just opened up very exciting new theoretical possibilities.

Science fans like me are incredibly lucky to live in the 21st Century, when science literally advances daily. My favorite site for following scientists' latest discoveries: ScienceDaily.com.

I’ve scarcely posted for over a year because I was busy working much of 2012. I worked 3 ½ months as a contractor programming in Ruby, Rails, Javascript, JQuery, Coffeescript, Backbone.js, Mongo, and HTML/CSS. Then I took a permanent position at a somewhat larger startup using those same tools, plus Postgresql and R.

I love programming. I began programming in COBOL in junior high school, but I got scared away from computers for the next 25 years after reading a book on the absurdly difficult Assembly language that made me feel programmers must all be mad geniuses.

I started dabbling with programming again in 2000 when I progressed from re-designing my company’s data schema to writing triggers and stored procedures (i.e., small programs that run inside a database). I dabbled with several languages — Java, PHP, C — before falling in love with Ruby in early 2006. I love that when a program fails it’s almost certainly because you — the programmer — blew it. The corollary is that when a program runs successfully, it’s because you wrote it correctly. It’s deterministic. It’s also powerful. I founded OptimalHome.com in 2001 and helped build a pretty amazing, fully functional prototype (covering just Silicon Valley) of an online real estate service that had some features still unavailable a decade later. But I lost my co-founder to an offer he couldn’t refuse (giving him an immediate fat paycheck and an 8-figure payday a few years later), and I couldn’t keep OptimalHome alive because I didn’t know Java. Ever since, I’ve wanted to understand the “full tech stack.” I’m not an expert programmer… not yet, anyhow. And I’ll certainly never be a genius. But I do enjoy programming and trying to become a better programmer every day.

I used to read the news more broadly and deeply. These days, my life is mainly spent on work and parenting, plus studying a little Chinese in my spare moments.

But when I’m programming, I often want to quickly grab a PDF reference document without searching through the directory tree. I want links to PDF files on Jasmine, RSpec, Rails, Ruby, Coffeescript, Underscore, JQuery, Javascript, Backbone, etc. all in one directory.

And when I have free time to watch videos, I’d like to quickly see a list of all available programming video files.

I’ve just created a (very ugly) webpage — RaisingMatureKids.com — where I’m posting chapters from the parenting book I began writing in January.

Raising Mature Kids is a work in progress that languished for many months while I threw myself into a more-than-fulltime programming gig. As I find time, I will clean up additional chapters and post them to the book’s website.

I was shocked to discover this week that my “htmls_to_pdf” Ruby gem had been downloaded 300 times. I released it just to practice releasing a ruby gem. I hadn’t expected anyone to find/use it.

Since it has been downloaded so many times, I figure others share my interest in wrapping multiple HTML pages into a single PDF file.

So I made some improvements this week (and fixed a bug that broke some of my sample code — caused by me changing the public interface… for shame!) and began adding tests. I also added more example files for downloading documentation and tutorials for Ruby, RSpec, Rails, Python, Coffeescript, Bash, and Backbone.js.

Since some would prefer to download the PDF files than install and run my gem, I will try placing a subset of the PDFs I’ve created on my website.

Please download once and save the files. Please don’t bookmark and download repeatedly.

Also, this is running on a small server slice, and some of these files are huge, so please be patient.

New research into Earth’s paleoclimate history by NASA’s Goddard Institute for Space Studies director James E. Hansen suggests the potential for rapid climate changes this century…

Hansen found that global mean temperatures during the Eemian period, which began about 130,000 years ago and lasted about 15,000 years, were less than 1 degree Celsius warmer than today. If temperatures were to rise 2 degrees Celsius over pre-industrial times, global mean temperature would far exceed that of the Eemian, when sea level was four to six meters higher than today, Hansen said.

“The paleoclimate record reveals a more sensitive climate than thought, even as of a few years ago. Limiting human-caused warming to 2 degrees is not sufficient,” Hansen said. “It would be a prescription for disaster.” …

…this research is consistent with Hansen’s earlier findings that carbon dioxide in the atmosphere would need to be rolled back from about 390 parts per million in the atmosphere today to 350 parts per million in order to stabilize the climate in the long term.

The deck is stacked against kids who live in poverty not just because their schools are on average worse than others, but also because of the circumstances of their lives when they leave campus.

It’s time that we admit that it isn’t just teachers holding back poor and minority students back. The problems are societal…

Let the 50 states disaggregate equality-related data by ethnicity, gender, and socioeconomic status, and let us rank the states and reward them for closing all the societal inequalities that are truly at the heart of our achievement gap….

Let’s have national benchmarks for equality in incarceration, equality in college enrollment, equality in health coverage, equality in income levels, employment rates, rates of drug addiction and child abuse.

My initial reaction was that reducing inequality — though a noble objective — is impractical because those who control America love inequality:

Our entire political economy has been systematically structured over the past 30+ years to maximize — not minimize — inequality. And now that the 1% (really the 0.1%) owns everything — the Supreme Court, Congress, the presidency, the corporations, the media, the cash — they’re grabbing everything they can, and there’s no way they’re giving a penny back without a fight.

When Warren Buffett said a few years back that there’s a class war and that his class is winning, he meant it metaphorically, based on massive tax cuts for the wealthiest Americans.

But now there’s literally blood on the Occupy movement who were systematically crushed nationwide for exercising their Constitutional right to peacefully protest. And journalists have been arrested for doing their Constitutionally protected jobs. New York City police (not to mention the feds and the military) have been running a nationwide J. Edgar Hoover-esque domestic spy operation since 9/11 and operate a many-tens-of-millions-of-dollars surveillance network in Manhattan jointly with the major banks (police sit side-by-side with bank personnel). They used it to spy on the Occupy movement. Our tax dollars are being used to spy on us to repress small-“d” democracy and efforts to shift the tax burden more toward the wealthiest Americans.

The rich and powerful — on their private jets and yachts — don’t care about educating ordinary Americans, let alone poor/minority Americans. The essay suggests a clever way to pursue a noble, wonderful ideal. But it’s diametrically opposed to everything America has stood for these past three decades.

But I then realized that even eliminating inequality would be only a small step in the right direction. The biggest problem with American education is anti-intellectualism:

That teachers can single-handedly teach our kids — regardless of their pre-school experiences or their lives outside the classroom — everything they need to know is absurd. So, too, is the corollary that teachers should be punished when kids from screwed up families and communities struggle and fail.

Consequently, factoring in socio-economic data would be a big improvement.

But there’s a far more important factor than class and income: motivation. Much of America has an anti-intellectual bias. Whereas students in countries like China look up to their successful, hard-working classmates, American students tease and bully “nerds,” “geeks,” “dorks,” and “teachers' pets.” We admire successful student-athletes but mock successful student-students. Many parents either don’t care about learning or don’t know how to encourage and support their children’s learning (as opposed to pushing them to get good grades, even by cheating and/or pressuring/threatening teachers).

Studies consistently find that a superb predictor of a child’s school success is the number of books in that child’s house. (We’re well over 1,000 books in our house, and I’m hoping the boxes of books stacked in my garage also count toward our total!) Number of books in the home is an even more powerful statistical factor than parental education. It serves as a good proxy for the importance parents place on education. And 27% of American adults tell pollsters they didn’t read a single book last year! (Since the data’s self-reported, the real percentage is probably even higher.) ¼ to 1/3 of American parents are effectively telling their kids that learning doesn’t matter.

So, even eliminating economic and social inequality in America would not solve American schools' critical motivation deficiency. I know because I’ve seen the children of extremely poor, illiterate migrant workers in Shanghai do exceptionally well in school. Shanghai’s school system scores even higher than Finland’s (and every other participating nation’s) on the PISA tests. Tavis Smiley and Cornell West visited quite a few children of migrant workers who live in dirty one-room shacks. Their children embraced and excelled in school because their parents and they placed such value on learning and because Chinese culture respects teachers. The walls of these children’s tiny “homes” were covered with school achievement certificates like wallpaper.

I suspect this “motivation gap” may explain the apparent success of early charter schools and their subsequent disappointing results. Early charter schools may have attracted the most pro-education parents, whose kids were most likely to embrace school and learning. As charter schools spread, their populations increasingly resembled those of ordinary public schools.

School should be about joyous learning, exploring, singing, painting, reading, dancing, and creating. Instead, most American schools are focused on filling in the right bubbles. Focusing on tests is a great way to suck the natural joy out of learning. And that’s toxic when combined with our society’s existing anti-intellectual bias.

The education gap in a nutshell: “In 2004, a National Endowment for the Arts report titled ‘Reading at Risk’ found only 57% of American adults had read a book in 2002… Many in the survey reported reading dozens of books and said they couldn’t do without them” USA Today.

Being raised by parents who read voraciously must feel completely different than being raised by parents who never read. Beyond the implicit message children receive watching their parents read (or not read), parents who read (rather than golf or watch TV) likely communicate better (thus building their children’s vocabularies) and read to/with their children, equipping them with valuable knowledge that helps them read.

This move reflects either criminal incompetence or abject corruption by the Fed. Even though I’ve expressed my doubts as to whether Dodd Frank resolutions will work, dumping derivatives into depositaries pretty much guarantees a Dodd Frank resolution will fail. Remember the effect of the 2005 bankruptcy law revisions: derivatives counterparties are first in line, they get to grab assets first and leave everyone else to scramble for crumbs. So this move amounts to a direct transfer from derivatives counterparties of Merrill to the taxpayer, via the FDIC, which would have to make depositors whole after derivatives counterparties grabbed collateral….

During the savings & loan crisis, the FDIC did not have enough in deposit insurance receipts to pay for the Resolution Trust Corporation wind-down vehicle. It had to get more funding from Congress. This move paves the way for another TARP-style shakedown of taxpayers, this time to save depositors. No Congressman would dare vote against that. This move is Machiavellian, and just plain evil….

Bill Black said that the Bloomberg editors toned down his remarks considerably. He said, “Any competent regulator would respond: “No, Hell NO!” It’s time that the public also say no, and loudly, to yet another route for running a drip feed from taxpayers to banksters.

If you have money in Bank of America, I would take it out tomorrow and move it to your local bank because BoA appears to have potentially trillions of dollars in gambling losses it has just moved to the same subsidiary that holds your bank account.

(Some, including U.S. Senator Bernie Sanders (I-VT), advocate moving money out of major banks for a different reason: to punish them for their predatory behavior. That’s a totally separate issue.)

The 2008 financial crisis could look like a hiccup if European defaults trigger massive credit default swap (CDS) payouts from major US banks.

Bank of America’s holding company — the parent of both the retail bank and the Merrill Lynch securities unit — held almost $75 trillion of derivatives at the end of June, according to data compiled by the OCC. About $53 trillion, or 71 percent, were within Bank of America NA.

What SHOULD happen if Bank of America suffers a multi-trillion-dollar gambling loss? The FDIC-insured portion of the bank should remain intact and regular depositors' accounts should remain untouched while the rest of the bank goes bankrupt, shareholders are wiped out, and other creditors fight for scraps in bankruptcy court. Those who placed winning bets with BoA would likely receive pennies on the dollar. And taxpayers would not contribute a nickel.

But Bank of America today did something that potentially puts taxpayers on the hook to bail out tens of trillions of dollars in idiotic BoA gambling losses. BoA moved tens of trillions of dollars of potential gambling losses into the same BoA subsidiary that holds massive quantities of FDIC-insured bank deposits.

Bank of America Corp. (BAC), hit by a credit downgrade last month, has moved derivatives from its Merrill Lynch unit to a subsidiary flush with insured deposits…

Moving derivatives contracts between units of a bank holding company is limited under Section 23A of the Federal Reserve Act, which is designed to prevent a lender’s affiliates from benefiting from its federal subsidy and to protect the bank from excessive risk originating at the non-bank affiliate, said Saule T. Omarova, a law professor at the University of North Carolina at Chapel Hill School of Law.

“Congress doesn’t want a bank’s FDIC insurance and access to the Fed discount window to somehow benefit an affiliate, so they created a firewall,” Omarova said.

The Federal Reserve and Federal Deposit Insurance Corp. disagree over the transfers, which are being requested by counterparties, said the people, who asked to remain anonymous because they weren’t authorized to speak publicly. The Fed has signaled that it favors moving the derivatives to give relief to the bank holding company, while the FDIC, which would have to pay off depositors in the event of a bank failure, is objecting, said the people.

I’m going to go out on a very short limb and presume one of Bank of America’s counterparties is Goldman Sachs. Years ago, Goldman helped cook Greece’s books to get Greece loans it had no business getting. Based on Goldman’s behavior, I would be shocked if Goldman didn’t then turn around and use its inside information to place massive bets (through credit default swap derivatives) against Greece’s debt. Merrill Lynch (now part of Bank of America) apparently was the stooge for many such stupid CDS bets that Bank of America is now on the hook for. If so, then Bank of America is on the hook to Goldman for tens of trillions of dollars if the European debt crisis explodes. And Goldman — worried BoA wouldn’t be able to pay off its gambling debt — has now forced Bank of America to taxpayers on the hook for the gambling debt.

If taxpayers are stuck with tens of trillions in derivative contract (gambling) losses, we’ll all soon look back at the 2008 banking crisis as “the good old days.”

Wall Street’s audacity to corrupt knows no bounds and the cooptation of government by the 1 per cent knows no limits. How else to explain $150 million of taxpayer money going to equip a government facility in lower Manhattan where Wall Street firms, serially charged with corruption, get to sit alongside the New York Police Department and spy on law abiding citizens….

The surveillance plan became known as the Lower Manhattan Security Initiative and the facility was eventually dubbed the Lower Manhattan Security Coordination Center. It operates round-the-clock. Under the imprimatur of the largest police department in the United States, 2,000 private spy cameras owned by Wall Street firms, together with approximately 1,000 more owned by the NYPD, are relaying live video feeds of people on the streets in lower Manhattan to the center. Once at the center, they can be integrated for analysis. At least 700 cameras scour the midtown area and also relay their live feeds into the downtown center where low-wage NYPD, MTA and Port Authority crime stoppers sit alongside high-wage personnel from Wall Street firms that are currently under at least 51 Federal and state corruption probes for mortgage securitization fraud and other matters.

In addition to video analytics which can, for example, track a person based on the color of their hat or jacket, insiders say the NYPD either has or is working on face recognition software which could track individuals based on facial features. The center is also equipped with live feeds from license plate readers.

According to one person who has toured the center, there are three rows of computer workstations, with approximately two-thirds operated by non-NYPD personnel. The Chief-Leader, the weekly civil service newspaper, identified some of the outside entities that share the space: Goldman Sachs, Citigroup, the Federal Reserve, the New York Stock Exchange. Others say most of the major Wall Street firms have an on-site representative….

The project has been funded by New York City taxpayers as well as all U.S. taxpayers through grants from the Federal Department of Homeland Security. On March 26, 2009, the New York Civil Liberties Union (NYCLU) wrote a letter to Commissioner Kelly, noting that even though the system involves “massive expenditures of public money, there have been no public hearings about any aspect of the system.”

Over and over, loyal, deserving employees with modest incomes have watched their planned retirement savings disappear because of corporate managers and pension industry consultants. Journalist Ellen Schultz has been writing about such shameful behavior for a long time, mostly in The Wall Street Journal. Now she has pulled together the copious, irrefutable evidence between the covers of a book. It is shocking, and demoralizing….

The book is crammed with heartbreaking anecdotes of retirees suffering (and in some cases probably dying) because of pension-related corporate greed. But the perpetrators have not been charged with any crimes.

…she writes, “What [GE CEO] Immelt didn’t mention was that, far from being a burden, GE’s pension and retiree plans had contributed billions of dollars to the company’s bottom line over the past decade and a half, and were responsible for a chunk of the earnings that the executives had taken credit for. Nor were these retirement programs — even with GE’s 230,000 retirees — bleeding the company of cash. In fact, GE hadn’t contributed a cent to the workers' pension plans since 1987 but still had enough money to cover all the current and future retirees.”

Then Schultz delivers the clincher: GE was indeed burdened by a pension plan — the plan for top executives. The obligations of that plan, for a minuscule number of individuals compared with the 230,000 lower-level retirees, totaled $4.4 billion and had drained about $573 million from the corporate treasury over the past three years.

I (finally) started Tweeting in August and have found Twitter to be a superb tool for keeping up with the ever-changing tech world. But Tweets seem pretty ephemeral, so I decided to write a script to archive my Tweets. Thanks to the “twitter” Ruby gem, it’s quite easy to save the past week’s worth of Tweets. But going back in time required a lot of annoying copy-and-paste (and translating quotation marks to “&quot;” and back again to make JSON happy). Anyhow, I’ve now collected here all my Tweets, some of which (esp. retweets) I think are very funny:

RT @KentBeck: my favorite idea from steve jobs is it’s the programmer’s job to adapt the computer to the user, not the user’s job to adapt to the computer (06 Oct)

Hank Williams Jr accuses ESPN of stomping on Free Speech, but that protects against govt. Compare Obama to Hitler and people can ignore you! (06 Oct)

Cucumber scenarios should not be a series of steps [saying] what a user clicks [but] should express what a user does github.com/cucumber/cucumber-rails/commit/f027440965b96b780e84e50dd47203a2838e8d7d/ (06 Oct)

Nobel winner: “There was a Swedish voice on the phone. I knew it wasn’t Ikea. I quickly realized the magnitude of it.” businessweek.com/news/2011-10-04/scientists-win-physics-nobel-for-dark-universe-discovery.html/ (04 Oct)

After installing 500GB HD, I figured I was set. But that encouraged me to download video and now I’m constantly struggling to free up space. (04 Oct)

If I pay my cable (and Internet) provider $150/month, why can’t I watch TV without being bombarded with ads? Commercial TV used to be free. (04 Oct)

RT @paul_irish: More on CSS shaders: adobe.com/devnet/html5/articles/css-shaders.html/ between this and CSS Filters (dvcs.w3.org/hg/FXTF/raw-file/tip/filters/publish/Filters.html/), the web is about to get so so so hot. (04 Oct)

RT @webtonull: A DBA walks into a NOSQL bar, but turns and leaves because he couldn’t find a table (04 Oct)

RT @DavidSacks: TSA patted down my 2-year-old and swabbed her hands for bomb residue. Our country has lost its mind. (07 Sep)

RT @gruber: Shorter Mike Arrington: “I sold my company (to a bunch of idiots) and expected to still maintain control over it.” (06 Sep)

RT @Nouriel: Berlusconi defense: I screwed the entire country for 10 yrs for free & no one cared. Now I screwed a few for money and they give me hell (04 Sep)

BN.com canceled my TouchPad order 2 ½ days after I placed it. Thanks for getting my hopes up and preventing me from buying elsewhere! (24 Aug)

Oops. That’s “classy family.” Note to self: Never squeeze in a last-second Tweet while running out door to take daughter to doctor’s office. (24 Aug)

Quiz: Your son plays for the NY Jets. Who do you cheer for? The Patriots, of course! Classfy family! nesn.com/2011/08/james-ihedigbos-family-still-rooted-for-patriots-when-he-played-for-jets.html (24 Aug)

RT @Nouriel: US has crumbling infrastructure & tons of laid off workers in construction. So we need a new fiscal stimulus to build infrastr & create jobs (19 Aug)

Alcohol’s good for my brain! Good thing I drank two beers while watching the (unstoppable) Patriots last night! ibtimes.com/articles/200678/20110819/moderate-drinking-alcohol-consumption-reduce-dementia-alzheimer-s-risk-study.htm (19 Aug)

RT @Nouriel: Thus gold, like US Treas yields, is pricing risk that we go into another deflationary depression & global financial meltdown, not inflation (18 Aug)

In theory, I’ve always liked the database agnosticism of Rails migrations. But, in practice, my migrations are PostgreSQL-specific because I usually fall back to writing SQL DDL statements since standard migrations can’t create essential database features like foreign key constraints and uniqueness constraints (which must exist in the db layer as well as the model layer or you’ll eventually have problems).

I thought of a solution that probably seems obvious to many developers, but I didn’t realize it till now.

From now on, I’ll write as much as I can using standard Rails migrations and then tack on Postgres-specific enhancements wrapped in a test that returns true if-and-only-if the database is Postgres. ActiveRecord makes it easy to test for the database adapter:

if ActiveRecord::Base.connection.adapter_name == ‘MySQL’

This embraces the same progressive enhancement principle advocated for adding JavaScript to web pages, for example.

This way, Rails migrations remain database agnostic yet also can add use proprietary database extensions. An added bonus: future developers can see the conditional code and adapt it for other databases if they wish.

Even toddlers in promotional posters wear skimpy clothes, vampy looks and makeup. At home later, the children will likely watch TV. Little ones, 2 to 11, average 32 hours a week…. During those hours, they’ll drink in ads for hair products and teen-siren TV shows, makeup and technology, much of it couched as “hot” or “sexy.” …[Other examples include] Vogue covers featuring small girls made up and posed like grown women, thongs and push-up padded bras for children as young as 6, Walmart’s line of 70 make-up products for girls 6 to 12 [and] skinnier-than-life Barbies targeted toward 3-year-olds. …[N]early a third of girls' clothing sizes 6 to 14 is “sexy.”

This angers me, but kids are being exposed to it, so parents need to understand and help their kids develop healthy attitudes.

“While saying no is a natural parental instincts, he says the optimal approach is to help their child understand why a certain TV show or piece of clothing is not OK. ‘You’d be surprised at how reasonable children can be when rules are accompanied by an explanation,’ he says. ‘Children are always learning. If they’re not learning from their mothers or fathers, they are going to learn from other sources.’”

“Thomsen thinks it’s vital that a young girl hears that their fathers think she’s a lovely, wonderful person. ‘That’s feedback that will maybe make her feel strong and resistant to other influences.’”

“she’s also guiding them through a process of analyzing things critically. "What do you think they’re trying to sell?” she asks when a model runs her fingers through her luxurious hair for a shampoo commercial. ‘Is it just shampoo?’"

“What receives praise matters, too. Instead of telling a child she’s smart or beautiful, he recommends praising traits like how nurturing she was to her doll or how hard she worked on her term paper.”

the compound is already in human clinical trials for cancer and eye disorders, and several hundred people have been exposed without major side effects. The new study revealed that squalamine can also disrupt a virus’s life cycle and prevent it from replicating in both tissue cultures and live animals.

The story of how it works is fascinating:

Squalamine is a positively charged molecule, so when it enters a cell, the molecule immediately sticks “like Velcro” to the cell’s inner membranes, which have negative charges, Zasloff said.

By doing so, squalamine “pops off” any positively charged proteins that were attached to the cell membrane—an action that does no harm to the cell, Zasloff noted.

When a virus invades a cell, it expects those proteins to be present on the cell membrane. Without them, the virus can’t reproduce.

“There is no other compound known to science that does this—this is a remarkable property,” Zasloff said.

Improving the ergonomics of chairs and other equipment increases productivity by an average 17%, based on a review of 40 studies of office workers published in 2008 in the Journal of Safety Research. Workers tended to have fewer musculoskeletal problems and a lower rate of absences and errors, the studies found.

I hear all this, “Oh! This is class warfare.” No! There is nobody in this country who got rich on his own. Nobody. You built a factory out there — good for you. But I want to be clear. You moved your goods to market on the roads the rest of us paid for. You hired workers the rest of us paid to educate. You were safe in your factory because of police forces and fire forces that the rest of us paid for. You didn’t have to worry that marauding bands would come and seize everything at your factory… Now look. You built a factory and it turned into something terrific or a great idea — God Bless! Keep a Big Hunk of it. But part of the underlying social contract is you take a hunk of that and pay forward for the next kid who comes along.

Steve Jobs is today considered perhaps the greatest CEO in many decades, but he was once fired by Apple, the company he founded!

Like Jobs and many other unconventional geniuses, Bill Belichick was fired. Belichick had the Cleveland Browns moving in the right direction till owner Art Modell announced he would move the team to Baltimore. Belichick inherited a 3-13 Browns team and, over four seasons, coached them to continually improving records of 6-10, 7-9, 7-9, and 11-5 (plus a playoff win). The following season, things started well, but Belichick’s Browns collapsed after news of the move to Baltimore leaked, and he lost his job. The renamed Ravens suffered three consecutive losing seasons after Belichick left. Even ignoring Belichick’s amazing post-Browns success, Belichick’s Browns record looks pretty good. But Art Modell (and most sports journalists) failed to appreciate Belichick’s football genius.

Many pay lip service to “thinking outside the box” but few firms actually behave as if they value and reward innovative, creative, multi-disciplinary people. Radical innovators often must build their own firms from scratch to breathe life into the wild visions in their minds.

“I’m not sure Apple even thinks about the competition,” Yamashita says. “They’re uniquely themselves without worrying about anyone else. When I worked for Steve there was little discussion about the competition. The aim was for us to be the most extreme version of ourselves.” …

“Apple is obsessiveness to the power of 10,” he says. “And you see it everywhere. It’s in things that are immediately visible, like the retail stores. They spend more on tenant improvements than anyone: stone floors, not wood floors; glass tables, not plexiglass. But it’s also in things you don’t ordinarily see: Apple has the most symmetrically laid out motherboard in the industry. They’re obsessive about the supply chain, obsessive about the design of the product, the packaging.”

Yamashita tells the story of Jobs’s walk-through before the opening of the first Apple Store. At the time, Apple was selling iMacs, the candy-colored computers that came in different “flavors.” Jobs walked into the store where the computers were lined up flawlessly on a table, took one look at the display, then ordered the computers taken off and the table turned upside down. Then he pointed at a seam on the bottom of the table and pronounced it “unacceptable.” Of course, customers would never see it. That wasn’t the point. The point was it was there. The point was not to allow an imperfection. The point was not to give up on the Apple dream.

“Apple has always been on an ongoing journey to be its best self,” Yamashita says. “Its marketing mission is to help Apple customers get the most out of their Apple products, to equip and enable their customers to be their best self, too. That kind of thinking has led to online tutorials, lessons at the Apple Stores, ‘gen­ius bars.’ What other company would hire 12,000 experts and then not charge customers a penny to talk with them?”

As Levin watched the progress of those KIPP alumni, he noticed something curious: the students who persisted in college were not necessarily the ones who had excelled academically at KIPP; they were the ones with exceptional character strengths, like optimism and persistence and social intelligence. They were the ones who were able to recover from a bad grade and resolve to do better next time; to bounce back from a fight with their parents; to resist the urge to go out to the movies and stay home and study instead; to persuade professors to give them extra help after class.

(I’m uncomfortable calling anyone “dumb,” but Tony Romo makes tens of millions of dollars to play a game and single-handedly threw away — through repeated acts of stupidity — a game his team had already won. All Romo had to do to seal the victory was not hand the ball back to the Jets. Instead, he gifted the ball to the Jets TWICE in the final quarter. He deserves to be called “dumb.”)

As a Patriots fan, I was thoroughly enjoying tonight’s Dallas Cowboys thumping of the Jets. Dallas was up by 14 points and had 1st-and-goal at the Jets' 2 yard line. Soon to be at least a 17-point 4th quarter lead. The Cowboys franchise had NEVER lost a 14-or-more-point 4th quarter lead. And Dallas is about to tack on at least 3 more. Game over.

Except I hadn’t accounted for the world’s dumbest man, Dallas quarterback Tony Romo, throwing the game away. When you’re up by two touchdowns in the 4th quarter and have an automatic chip-shot field goal — even if you’re sacked — 99% of your focus should be on not turning the ball over. Don’t worry about scoring a touchdown if there’s any risk involved. Throw the ball at the hot dog vendor or curl up in the fetal position. Just don’t turn it over. Instead, Romo decides he’s a running back and dives straight into the teeth of the Jets defense, immediately coughing the ball up at the 1-yard line. Jets get the ball back and score, making it a 7-point game. Then, after the Cowboys fail to move the ball, they try to punt and somehow forget to block the Jets defender in the very middle of the field… with the shortest path to their punter. It’s like Moses parted the Red Sea for the Jets guy to put his hand on the ball before the punter could kick it. Punt blocked and run back for an easy touchdown. Tie game.

OK, so you’ve blown a two-touchdown lead. But it’s still a coin flip. Dallas has a 4th-and-1 and punts it. I would have gambled on 4th down because Dallas had lost its top three cornerbacks and the Jets were driving easily on their replacements. Going for it on 4th seemed a worthwhile gamble. But Dallas punts and eventually gets the ball back. But then Romo throws a short completion… to the Jets' Darelle Revis! Romo throws the ball to a double-covered receiver while the league’s best cornerback stands between Romo and his receiver. Are you kidding me, Tony? Have the bookies threatened your family or something? Or are you just the world’s stupidest person?

And I haven’t even mentioned the foolish delay-of-game penalties. Or the critical second-to-last play where the Cowboys needed about 30 yards but Tony wasn’t even paying attention when his center snapped the ball to him. While Romo was looking elsewhere, the ball clanged off his chest, killing whatever hope Dallas still had of kicking a tying field goal. And this was, I believe, after a Jets timeout! How does a quarterback not know the snap count after a timeout? Amazing. The Jets had no business winning that game. Romo served it on a platter.

As Carl Sagan once famously said, “It is of interest to note that while some dolphins are reported to have learned English – up to 50 words used in correct context – no human being has been reported to have learned dolphinese.”

…[A]ccording to information theory, dolphin communication is highly complex with many similarities with human languages, even if we don’t understand the words they are saying to one another.

Information theory was developed in the 1940s by the mathematician and cryptologist Claude Shannon, mainly to be applied to the then-burgeoning technology of telecommunications. It operates on the knowledge that all information can be broken down into ‘bits’ of data that can be rearranged in myriad ways. George Zipf, a linguist at Harvard, realized that language is just the conveyance of information, and therefore could be broken down too.

Think of all the different sounds human beings make as they speak to each other, the different letters and pronunciations. Some, such as the letters ‘e’ and ’t' or words such as ‘and’ or ‘the’ will occur far more frequently than ‘q’ or ‘z’ or longer words such as ‘astrobiology’. Plot these on a graph, in order of the most frequently occurring letters or sounds, and the points form a slope with a –1 gradient.

A toddler learning to speak will have a steeper slope – as they experiment with words they use fewer sounds but say them more often. At the most extreme a baby’s babble is completely random, and so any slope will be nearly level with all sounds occurring fairly evenly. It doesn’t matter which human language is put through the information theory test – be it English, Russian, Arabic or Mandarin – the same result follows.

What is remarkable is that putting dolphin whistles through the information theory blender renders exactly the same result: a –1 slope, with a steeper slope for younger dolphins still being taught how to communicate by their mothers, and a horizontal slope for baby dolphins babbling. This tells us that dolphins have structure to how they communicate.

Meanwhile, another feature of information theory, called Shannon entropy, can tell us how complex that communication is…. Write down 100 words on one hundred pieces of paper and throw them into the air and they can be arranged in myriad ways. Impose rules on them, such as sentence structure, and your choices automatically narrow. It is a bit like playing hangman; you have a five-letter word where the first letter is ‘q’, so the rule structure of English necessitates that the second letter is ‘u’. From thereon there is a limited number of letters that can follow ‘qu’ and so you may have ‘que’ or ‘qui’ or ‘qua’ and you can predict that the word is ‘quest’ or ‘quick’ or ‘quack’. Shannon entropy is defined as this application of order over data and the resulting predictability of that order.

“It turns out that humans go up to about ninth order Shannon entropy,” said Doyle. “What that means is, if you are missing more than nine words then there is no longer a conditional relationship between them – they become random and pretty much any word will do.” In other words, there are conditional probabilities, imposed by the rule structures of human languages, up to nine words away.

Doyle has analyzed many forms of communication with information theory, from the chemical signals of plants to the rapid-fire radio transmissions of air traffic control. How do dolphins fare? “They have a conditional probability between signals that goes up to fourth order and probably higher, although we need more data,” said Doyle.

In test-driven development (TDD), we make each test fail before we write the code that makes it pass. It’s a great practice because it helps you spot bugs almost instantly (esp. if you use “autotest” or a similar program to automatically run your tests every time you modify any code or test).

I expected my test to fail by overwriting the existing file and not raising an error. Instead, it passed — by raising an error when I hadn’t expected one since I hadn’t yet written the code to prevent overwriting existing files.)

To see the error, I ran:

it "won't overwrite an existing file" do
FakeFS do
File.open(house_appraisal_downloader.appraisal_dir + '/test_record.html', 'w') { |outf| outf.write("abcde") }
house_appraisal_downloader.save_as('test_record.html')
end
end

But I had forgotten to add , 'w' as the second parameter of File.open!

So thank you yet again, TDD. Writing failing tests before writing the code to make them pass is a brilliant way to catch errors immediately. It simultaneously builds a large test suite that documents your code and immediately identifies future code regressions.

In a 2010 study, James K. Harter and colleagues found that lower job satisfaction foreshadowed poorer bottom-line performance. Gallup estimates the cost of America’s disengagement crisis at a staggering $300 billion in lost productivity annually….

In one-third of the 12,000 diary entries [we collected], the diarist was unhappy, unmotivated or both. In fact, workers often expressed frustration, disdain or disgust. Our research shows that inner work life has a profound impact on workers’ creativity, productivity, commitment and collegiality. Employees are far more likely to have new ideas on days when they feel happier. Conventional wisdom suggests that pressure enhances performance; our real-time data, however, shows that workers perform better when they are happily engaged in what they do….

Workers’ well-being depends, in large part, on managers’ ability and willingness to facilitate workers’ accomplishments — by removing obstacles, providing help and acknowledging strong effort. A clear pattern emerged when we analyzed the 64,000 specific workday events reported in the diaries: of all the events that engage people at work, the single most important — by far — is simply making progress in meaningful work.

As long as workers experience their labor as meaningful, progress is often followed by joy and excitement about the work. “This time it looks good! I feel more positive about this project and my work than I’ve felt in a long time,” one programmer wrote after she’d completed a small but difficult task. This kind of rich inner work life improves performance, which further supports inner work life — a positive spiral.

…Of the seven companies we studied, just one had managers who consistently supplied the catalysts — worker autonomy, sufficient resources and learning from problems — that enabled progress. Not coincidentally, that company was the only one to achieve a technological breakthrough in the months we studied it.

The vending company at Rye Playland robbed me a few weeks back. Our son wanted cold water, so we shelled out $3 for bottled water, but a Pepsi bottle rolled out. No one in our house drinks soda, so the $3 was wasted and our son didn’t get any water. (We weren’t going to waste another $3 playing vending machine roulette!) I called the vending company and left a complaint. Never heard back.

And a few days ago, I shopped at a Stop & Shop (in Meriden, CT while returning from a vacation in Boston) and found $18 lying on the floor. I picked it up, held it aloft, and immediately shouted as loudly as I could, “Did anyone lose this money?” No one claimed it. But then a clerk grabbed the money out of my hands and handed it to Customer Service without asking my permission.

My guess is that one of two things will happen. Most likely, no one will claim the money and the store will keep it. Alternatively, one of the many people who heard me shout about the lost money may go falsely claim it. I doubt the person who lost it will realize they lost it, know where they lost it, return there, not find it, and then go to Customer Service. Even if they do, some fraud may have already claimed it or whoever is working at the customer service desk when the rightful owner returns might not know about the money or believe them.

Given the very low probability the money will be returned to its rightful owner, don’t I have a stronger claim on that money than the Stop & Shop? Shouldn’t they have taken my name and address and mailed it to me if no one claimed it? What ever happened to the “finders keepers” rule we all learned on the playground? $18 won’t pay the mortgage (or even the groceries we bought that day), but I feel like the Meriden Stop & Shop robbed me.

Posted by James on Aug 31, 2011

About Me

I love programming (Ruby, Python, Javascript), data analysis, being a dad, and learning how the world works. More…