Our engineering team has kicked some serious butt lately. We dropped a release candidate this morning for the current suite of applications in our growing library. JumpBox simplifies the deployment of Open Source server applications reducing what has traditionally taken four hours to setup down to a thirty second task. Our virtual appliances are designed to make life easier for IT people everywhere and bring these powerful and complex applications within the reach of non-technical users. The following applications are now feature-complete and quarantined for final fixes before we ship:

What can we possibly learn from butterflies and snails? The science of Biomimicry posits that we can observe naturally-occurring structures and processes formed through evolution, distill the mechanisms at work and apply the lessons towards solving modern-day problems. There’s an excellent podcast here with Janine Benyus, a woman who has dedicated her life towards extracting insights from things like the cell structures of butterfly wings the calcification mechanism in snail shells and applying them towards development of stealth aircraft and efficient plumbing systems. It stands to reason that with 4BN years of natural selection, evolution already has the answers to some tricky problems.

There are two fields in the space of Artificial Intelligence that take different approaches to solving problems but incorporate this same concept that we can emulate nature with software to solve complex problems. One attempts to emulate evolution (genetic algorithms) while the other emerging one is now attempting to emulate our brain function (called Hierarchical Temporal Memory Systems).

I spent the entire day yesterday doing nothing productive for our company but instead reading and listening up on these two fields of study and I’m blown away by the possibilities of each. I can’t possibly do justice to the explanation of either but here’s what I found interesting:

Genetic Algorithms – synonymous with the term evolutionary machine learning, this field has been around awhile. This amazing podcast with David Fogel explains his work with genetic algorithms in building a checkers program that learned to play at a grandmaster level with zero instruction on what the goal of the game was or how to win. It’s one thing to build a program and give it explicit instructions for how to evaluate moves and choose the best one- it’s an entirely different prospect to leave it untrained, turn it loose and tell it only the number of points it earned after x number of games. They essentially conducted Darwinian evolution with different virtual checker players using a program that ran 6mos on a Pentium II Windows NT machine and it learned to play on its own and ended up beating a grandmaster.

Granted the game of checkers is well-defined with clear rules and played in a bounded space (ie there’s no random external forces to account for like pieces getting knocked off the board by accident), but the mind-blowing implication of this story is that we no longer have to explicitly program a computer with the methods we know for solving problems, we can simply give it the problem and tell it when it’s doing well.

Hierarchical Temporal Memory Systems (HTMs) – this emerging field was started by Jeff Hawkins, the guy who created the Palm Pilot and wrote the book On Intelligence. It takes the approach of emulating the structure of the human brain (specifically the neocortex) with the idea that “our brains do things like facial recognition extremely well while the most powerful traditional computers cannot so there must be something to learn.” It breaks learning down to the idea that it’s based on patterns perceived in close temporal contiguity (back to back) and this idea that the storage mechanism consists of these hierarchical nodes that store different aspects of complex information at varying levels. For instance a computer running the Numenta software that’s hooked up to a camera and shown 100 instances of various dogs will begin to know the characteristics that make a dog (fur, four legs, tail, typical shape, etc). Information about the fur and the angles of the body are stored at the lowest nodes while concepts of body parts and the holistic notion of the essence of what makes a dog a dog, at the higher nodes. I haven’t gotten my head entirely around this concept but there’s an awesome six-page article here by Hawkins that explains this stuff and if you’re brave enough, there’s a longer white paper on his company site here that digs even deeper.

They have an interesting business proposition in that they have released the HTM software and API’s free for people to use as the engine for powering apps like facial recognition and intelligent network routing programs. They’re basically selling the raw “neocortex computing fabric” that anyone can use to train for any application. Combine this with the idea of utility computing and running your app on a virtual infrastructure like Amazon EC2 and S3 and you have the ability as a high school kid to build your own supercomputer with any world-changing application you can dream up.

In my high school physics we built solar cars out of balsa wood and tried to see who could build the fastest one- it was fun but in reality the skills we learned from those experiments had limited applicability. The forward-thinking high school physics teachers of today should be ditching the balsa wood and training their students on the concepts of genetic algorithms and HTMs and using them to conduct the equivalent experiments of the solar car challenge only with real problems. These concepts if grokked by youngsters now can be applied to solving hairy problems that will undoubtedly confront us in the next generation. Pollution, environmental change, contagious disease epidemics, harmful drug interactions, security threats, energy and vehicular traffic routing, diminishing energy resources- for all the daunting problems that could wipe out civilization, there’s extremely promising problem solving tactics emerging. Whether profit or non-profit, the important companies of tomorrow will be the ones that learn to capitalize on these technologies. Let’s hope that the high school physics teachers out there are listening and realize how important they are. My mother is 3wks away from finishing a 40yr high school teaching career – for they pay and the beauracracy that public education teachers put up with, I have to imagine the ones that stick it out are the ones that realize the value and ripple effects of what they do. And let’s hope that our Federal government too will recognize the importance of high schoolers becoming excited about this stuff now and start to prioritize expenditures accordingly.

When you’re the buyer in a transaction, you win when there’s multiple sellers and that are highly-motivated while you remain unattached to the transaction’s outcome. The converse is true when you’re the seller.

If there’s something you’re looking to buy but the immediacy of the purchase is not important, you can establish the above conditions through using RSS and persistent searches and simply being patient. Craigslist and eBay both have the ability to create a search for an item and persist it over time by monitoring an RSS feed.

For instance if I search for “guitar” in Phoenix on Craigslist it currently returns 845 results. I can whittle those down by specifying that I’m looking for a Gibson that’s priced under $300. That search returns a more manageable set of ten results. But what if none of those is right? There’s a link to a unique RSS feed for this search at the lower-right corner of the results page. By subscribing to this feed, it’s like having a college intern sit at a computer 24/7 running this search and notifying me when a new result is found. I have a separate category in my Bloglines called “Alerts” that’s specifically for the purpose of collecting notifications on persistent searches like this one.

The eBay equivalent of is to use their “Favorite Search” feature and have the results delivered via email. The only problem is that email is not the ideal way to consume these notifications- if you have many from multiple sources, you have to go into each system and manage them there rather than just dropping a feed.

The more useful approach that is not publicized on eBay is a free service called RSSAuction.com. It lets you setup the same search but delivers nearly real-time results (rather than daily results) available via RSS and you don’t have to deal with subscribing and unsubscribing to emails.

The reverse of the buying scenario above is the notion that as a seller, you’re best off selling an item when you don’t actually need to. You could take the time to list a bunch of items that you’d be willing to sell on all the various listing services but the reality is that’s a lot of work if you’re not actively trying to sell something. I came up with a concept I called that I think would become very popular with the “yardsale junkie” type and could unlock a whole new market of things that people aren’t actively trying to sell but are willing to part with for the right price. If you want to be a part of developing that project, contact me through the Cambrian House project link.

This is an analysis of the various options available for finding new music. As with anything the tools and techniques I’ll put forth are neither inherently good nor bad. If you use these tactics to steal music- shame on you. If you use them to get turned on to a bunch of new albums (some portion of which you end up buying) and it translates to more music purchases than would have occurred otherwise- then good for you (and good for the musicians). I’m simply going to explain the stages involved and the tools available.

The way I see it, from the time you first hear a new song that grabs your interest to the time that you plunk down money for it, there are three discreet phases that occur in this sequence:

These steps can be folded into one sitting but the chain is always the same in that you
a) discover the new music
b) listen to it more and determine that you like it enough to
c) go about purchasing it.
If we acknowledge these are the three stages we can examine which tools work best for each:

Discovery

Think of the situations in which you are most likely to discover a new artist- listening to the radio, having a friend play a song that you like, catching an opening act at a concert that surprises you… Of the ones that you have some control over, here are some thoughts on how to maximize your exposure to music you will like:

Traditional Radio – this is all but worthless for music discovery unless you’re hoping to discover Nickelback’s newest single or the next YHBTIM nomination. I stopped listening to FM radio two years ago and felt immediately smarter for cutting out the commercials and radio show banter that cluttered my daily drive. I also found that removing my antenna gave my (plus it made for a great shishkabob roasting implement on our last camping trip).

Satellite Radio – I had Sirius Radio for awhile and it was better than FM for the absence of commercials but even the programming on that got stale eventually. I never did try XM radio but I know some people swear by one or the other. At $13/mo I eventually decided for the limited amount of time I spent in the car and the extended hours I spent at the computer, that money was better spent on a Rhapsody subscription being able to listen to any band on demand. I’ll discuss the iPod radio place-shifting concept for your vehicle below.

Internet Radio – this is a great way to discover new bands at work. Shoutcast servers broadcast audio streams that can be consumed by iTunes, Winamp and Windows Media Player for decent quality audio grouped by different genres. There’s few if any commercials and quality of playlists varies station to station but I’ve found to be much more variety and originality than traditional or satellite radio.

Indie Station Podcasts – Garageband.com maintains a huge listing of indie radio station podcasts that you can subscribe to via iTunes so it automatically grabs the latest MP3’s. This is somewhat a “needle in a haystack” smattering of mostly crap but with some real gems to be discovered. If you’re the type that likes to hit a bunch of yardsales and hunt through stacks of mostly junk because you can somtimes discover that obscure gem, this is probably for you. But if you want more “signal than noise” you’re probably better off finding a couple good Internet radio stations.

iPod Place-shifted Internet Radio – Now for some controversy. So you can listen to Internet radio while you’re at your computer but you can also rip the audio streams to your iPod and listen to Internet radio in the car. I know some people will say this is cheating and this is where it comes to how you use this tactic- if you use it like a Tivo to truly time/place-shift your Internet radio, then I believe it’s a valid use. If you’re using it to steal music which you listen to repeatedly but never buy, then boo-hiss on you. The quality is going to be degraded at an avg of 96kbps and you’ll most likely get streams that chop off the beginning and ends of songs but if you can put up with those nuissances, this is a great way to avoid crap radio stations in the car and discover new stuff. Stream ripper is an open source plugin to Winamp that let’s you rip a shoutcast audio stream and store it on your iPod. Pandora’s Jar is a windows app that will rip a pandora station to your hard-drive and allow you to store it on your iPod. I’ve not used Pandora’s Jar. Whether you consider these tools “Tivo for Internet radio” or instruments of a thief, you should be aware of their existence.

Online Discovery Services – There are a ton of services online that can be divided into two categories:

Audio-based – Pandora and Last.FM are by far the two most popular though they use different methods for making recommendations. There is an excellent blog post that describes the essence of the difference in recommendation engines. Basically Pandora draws upon what is called the “Music Genome Project,” essentially a massive database of music classified by objective characteristics of the music itself (ie. tone, rhythm, timbre, etc). It’s a flash-based app that runs in your browser allows you to seed it with music you like and then give thumbs up/down as it makes recommendations to tune it to your personal preferences. Last.FM comes from the other angle making recommendations based on what’s called collaborative filtering, or purely by comparing listening preferences of people that share your likes. It doesn’t know anything about the qualities of the music itself. Last.FM is a client-based app that runs on Mac or Windows. Both produce totally different recommendations and can turn up some real gems based on your taste in music. And both have nifty widgets and online exposure tools if you run a blog to be able to broadcast your stations and listening history (see the audio section in the left side bar of this site). Mog and iLike are two relatively new entrants to the music recommendation game, neither of which I have tried.

Visual-based – among the visual mapping tools for music discovery are Live Plasma and Musicmap. Live Plasma seems to have come a long way and when I just tested delivered a lot more recommendations with meaningful associations expressed via their slick interface. It also appears to handle movies now as well. Musicmap looks more like someone’s hobby site. Both let you branch indefinitely following nodes upon nodes of recommendations with the ability to dig in for detailed album info on amazon.

So those are the basic methods for discovering new songs. Next we’ll talk about how “vet” the music to determine whether it’s worth buying.

Vetting

Online – You’ve got two flavors, paid and unpaid services for exploring albums. You definitely get what you pay for in this realm. Although Rhapsody just raised the rates of their service ($10 -> $13) I still find it completely worth the price.

Paid – all-u-can-eat music services like Rhapsody and Yahoo Music are great for when you hear a song and snag a name and want to dig into the rest of the album to see if it was a one-hit-wonder or if it’s truly representative of the rest of the album. I haven’t used Yahoo’s service but Rhapsody is great. If you’re on a mac you’ll want to use the YottaMusic.com player. It’s a killer ajax-based app that uses your Rhapsody account credentials to play music via the browser. It used to be pretty flaky crashing the browser but seems to have become much more reliable recently and I’m told by the Yotta team that the issue was actually a problem with the Rhapsody plug-in. I have no idea how these guys are making money but I would hate to see them go away at this point since they have such a valuable service.

Unpaid – most of the discovery services will link to the album on Amazon and take a portion of the sale via the affiliate program. Amazon has short audio samples for most albums playable via browser but the quality is crap and you don’t get the flavor of the album, just a taste of each song. There’s a new one I heard about recently called Musotik but it seems fairly amateurish and had audio for maybe half the songs I searched.

Physical Stores– it probably doesn’t deserve mentioning but there’s the obvious option of visiting a Best Buy or Borders for it’s listening station to sample music from the album. Depending on how obscure the music you like is, this may or may not be an option.

So now that you’ve found the new sounds and determined which are worth purchasing, let’s look at the options for making that transaction.

Purchase

iTunes– with the classic $.99 immediate gratification of purchasing legal music (and now the slightly more expensive option for purchasing non-DRM’d music), iTunes is by far the most common option for purchasing music online. Walmart entered the online music game and in their typical fashion tries to undercut the incumbent by selling at $.88/song but Walmart’s site was down when I checked it just now and they’re also a scourge to humanity so you should pay the extra and give it to a kickass company like Apple. There’s a Russian-based outfit called AllofMP3.com which supposedly has high-quality, dirt-cheap music but I understand there’s some type of sanction against them and Visa and Mastercard dropped them so they’re probably not honoring the licensing deals with the labels.

Local used stores– Let’s not rule out the trip to the local used record store that can provide both excellent recommendations from knowedgeable workers for discovery as well as second-hand copies of music you’re looking for (which is often cheaper than the online version if you are able to find it).

Mail-based clubs– there’s also music clubs like BMG. I’ve never used one of these but my brother built a massive CD collection fairly cheaply a few years ago by subscribing to one. The trouble is then you have a bunch of CD’s lying around.

Conclusions

There’s a lot of ways to expand your musical horizons. I find that listening to the same music repeatedly can put you in a rut and finding that fresh new band can propel you out of it. I also tend to associate the various chapters of my life with albums for some reason. I have no idea if musical variety is correlated with enhanced creativity but I would suspect it is. One other service that doesn’t fit into any of the above categories but that I’ve found hugely valuable for discovering music indirectly is the OnTour widget. They make both a Mac and PC version and it helps you find upcoming concerts in your area by scouring your iTunes library and then matching against a tour database and displaying the shows in your area of the bands you follow. My thoughts on the controversial ripping tools: if you use them as Tivo for Internet radio and doing so lets you discover more bands and buy new music you wouldn’t have bought otherwise, then it’s newfound revenue for the musicians and labels. If you disagree, cheers to having your own opinion and judgement. If you use other music discovery services or can suggest other methods for getting exposure to new music, please share.

Here’s a simple method I recently discovered for shrinking the size of PDF’s on a Mac. It was able to convert a 5MB PDF down to 800k while maintaining perfectly readable quality. The key is to re-save the PDF applying a quartz filter to compress the images. Here’s the steps to make it work:

Open your Colorsync Utility, click on the “Filters” tab and duplicate the one called “Reduce File Size.”

You’ll want to bump up the quality a bit from the default on that preset- I found the following settings to be about right to achieve a 1/6th reduction in file size while preserving readability:

Now close out of the colorsync utility and open up one of your bloated PDFs. Choose the File > Save As option and on the save dialogue apply the quartz filter you just set up.

This has been helpful in shrinking down PDF’s that come out of my scanner. The document scanner I have (Canon MP830) is neat because it has an automatic document feeder and the ability to turn a big document into a PDF on my desktop by pushing a single button, but for whatever reason the lowest resolution setting still produces these massive files. I looked around and found various open source programs that would do the same thing but the quartz filter mechanism built into Mac OS X works perfectly and requires no extra software. Thanks to the good people on this forum for pointing out this handy technique.