Thursday, June 30, 2011

"Unlike ordinary passive cables that can be used at lower data rates, the unprecedented speed of the new Thunderbolt technology places unique demands on the physical transmission media," according to Gennum's website. "The GN2033 provides the sophisticated signal boosting and detection functions required to transfer high-speed data without errors across inexpensive Thunderbolt copper cables."

As the Ars Technica review notes, this bears a striking resemblance to the early days of FireWire.

I was rather startled to realize, the other day, that it's been 4.5 years since Jim Gray, the greatest computer programmer of our time, was lost at sea.

Recently, Professor Joe Hellerstein of Berkeley published a technical report looking back on the amateur search efforts that were organized in the days after Gray's disappearance:Searching for Jim Gray: A Technical Overview.

The report discusses the work of Gray's friends and co-workers, who hoped to aid the search:

As we learned, real-time search for boats at sea is not as simple as getting a satellite feed from a mapping service, or borrowing a private jet.

As if even those tasks were simple!

Moreover, once they did conduct their image analysis, they learned that:

Our conclusion is simply that the ocean surface is not only very large, but also very empty.

The report also contains an interesting appendix speculating as to why the EPIRB on Tenacious never sounded an alert, and wonder whether the government might at some point consider:

mandate the installation of maritime safety technology in a failsafe way, as we have done with other technologies like automobile airbags. It is both possible and inexpensive (relative to the cost of a boat) to require EPIRB-like technology to be integrated into boat construction.

The paper is quite interesting, and quite different from the typical computer science technical report. I enjoyed it; you probably will, too.

All of a sudden, it seems like you can't go anywhere without bumping into a discussion of how SSDs are changing the world. Every conversation seems to be about IOPs, wear-leveling, non-rotational optimization, SSD benchmarking, or MTBF and duty cycles.

An abbreviated selection of some of the interesting recent activity:

Robin Harris digs into some of the details about block-level deduplication on SSD devices. Specifically, there is a discussion about the tradeoffs that storage vendors are making: "minimizing capacity use and maximizing data availability are conflicting goals", as Harris observes. The concern is that, if a storage device "de-duplicates" two blocks which the higher-level file system intentionally wrote redundantly, the underlying storage device may be undoing the file system's attempt to avoid a single point of failure. As David Rosenthal puts it:

File systems write the same metadata to multiple logical blocks as a way of avoiding a single block failure causing massive, or in some cases total, loss of user data. An example is the superblock in the UFS (Unix file system). Suppose you have one of these SSDs with a UFS on it. Each of the multiple alternate logical locations for the superblock will be mapped to the same underlying physical block. If any of the bits in this physical block goes bad, the same bit will go bad in every alternate logical superblock.

As several of the comments to Harris's article note, it's not clear that this interaction between the filesystem and the underlying storage system is unique to SSD; modern storage vendors have a variety of complex layers of logic which may or may not interact cleanly with the file system's algorithms. Meanwhile, Harris gets in touch with one of the storage vendors to discuss the details of error detection and error correction on SSD storage devices, and posts some interesting responses. One thing that is clear is that modern storage devices have become fantastically complex:

Not only do SandForce SSD Processors employ ECC protection enabling an UBER (Uncorrectable Bit Error Rate) of greater than 10^-17, if the ECC engine is unable to correct the bit error RAISE will step in to correct a complete failure of an entire sector, page, or block.

On the High Scalability blog, Todd Hoff highlights a recent presentation from the O'Reilly Velocity Conference by Artur Bergman of Wikia, talking about their use of a high-end all-SSD storage system. Jeff Darcy responds with the position that RAM, SSD, and disk should be thought of as a storage hierarchy, and arranging the levels of storage carefully is likely to lead to better efficiency.

Saturday, June 25, 2011

Instant answers. New sources of knowledge. Powerful tools—all for free. In just 13 years we’ve built a model that has changed the way people find answers and helped businesses both large and small create jobs and connect with new customers.

The FTC's preparations to subpoena Google are the first concrete signal that the agency's commissioners have decided there is enough evidence to move forward with a formal investigation. The probe is expected to take a year or more to unfold, and it won't necessarily lead to any charges.

Thursday, June 23, 2011

Connection-based means that you have a definite partner. You aren't just sending your data out randomly, you're sending it to the program on the other end of the connection.

Streaming means that your data is delivered in-order. Byte 1 will be delivered before byte 2, which in turn is delivered before byte 3.

Reliable, because the TCP/IP implementations take over a number of incredibly important details which each application would otherwise have to build on its own.

Full-duplex, meaning that both ends of the connection can be sending and/or receiving at the same time; the network does not mandate the rules about who can send or receive when.

Under the covers, TCP/IP takes the data that you give it, chops it up into packets, check-sums the data, addresses and routes it to its destination, acknowledges delivery, and automatically re-transmits lost or damaged packets.

It also automatically and adaptively detects congestion on the network, and controls the flow of data to ensure fair use of the network resources.

However, there is one thing that TCP/IP doesn't, and, can't, provide: the network is not of infinite capacity. There is some amount of sent-but-not-yet-received data which can be outstanding, but when you hit that limit, no more data may be sent.

Therefore, if you are not careful when you are designing your application-level protocol, it is possible to fall into a simple trap:

On my end, I start sending data to you. I keep sending and sending, and at some point all my buffers fill up and so my write() call blocks, waiting for you to receive some of that data.

However, on your end, you are doing the same thing: you keep sending and sending, and at some point all of your buffers fill up and so your write() call blocks, waiting for me to receive some of that data.

I'm never going to receive your data, because I'm blocked writing.

And you're never going to receive my data, because you're blocking writing.

This is commonly called a "write-write deadlock".

Unfortunately, this is the sort of error that you may not notice until your application has been in production for months or years, particularly if your local testing is performed on machines which are directly connected on local-area networks with large internal buffers.

There are various ways to avoid this problem, such as having programming conventions which ensure that each partner always reads data in preference to writing data, and, when reading data, programs always read as much data as is available before blocking or writing more data.

But the best way to avoid these sorts of problems is to be aware of them while designing and analyzing your application level protocols, so that you are careful not to build programs which fall into this trap.

Tuesday, June 21, 2011

David Weekly has written an interesting short article entitled: An Introduction to Stock & Options for the Tech Entrepeneur or Startup Employee, and has self-published it on Scribd and Amazon.

It won't instantly qualify you for an MBA program, nor, as he notes, should it be the basis of any legal decisions you make, but it's a clear, well-written, and thorough guide that will make you feel much less helpless and useless should you ever be lucky enough to be part of a venture capital-funded startup.

I wish I'd known all these things when I was 25, when it actually mattered. Of course, the bottom line is still that the investors and executives will be the ones that get rich, while you, the engineer, will do the hard work of designing and building the software, but at least you'll start to be able to understand why they are getting rich and why they keep nattering on about stuff that doesn't seem to matter instead of paying attention to running the company (which you foolishly thought was what they were supposed to be doing).

The technical program information for the 2011 USENIX Annual Technical Conference appears to be online now. Although the videos are restricted to USENIX members, access to the other technical information is available to all -- kudos to USENIX for making this information available to the broader community!

This conference has a fairly broad scope, with activity in many of the currently hot areas of computing:

Virtualization

Cloud Computing

Storage systems

Security

and more.

Among the talks that immediately jumped out at me as intriguing were these:

Imagine that you have been given the task to design a shipping-container cluster containing 1000 server-class computers. The container provides the necessary power and cooling, you have already chosen the servers, and now you must choose a network to connect the servers within the pod.

The Design and Evolution of Live Storage Migration in VMware ESX, Ali Mashtizadeh, Emré Celebi, Tal Garfinkel, and Min Cai, VMware, Inc. -- my co-worker Dave Ackerman was really impressed by this work by the VMWare team.

Building a High-performance Deduplication System, Fanglu Guo and Petros Efstathopoulos, Symantec Research Labs

This is just a sampling of what was discussed at the conference. It looks like there was a whole section of the conference devoted to advanced techniques for capturing and replaying system activity under the control of debuggers; another intriguing section had to do with improved scheduling techniques for things like multicore systems, non-uniform memory systems, and GPU-rich systems.

It looks like it was a very interesting conference, and I'm looking forward to digging into it in more detail.

In fact (dare I say it? this is a refereed computer science paper, after all!), the paper is exciting and suspenseful, almost heart-racing.

Unless you've been catatonic for the last decade, you're undoubtedly familiar with the basics of these operations; like most IT professionals, you probably get called in about once a week to untangle your neighbor or other associate from the mess they've stepped in. The paper sums it up succinctly:

The most common form of scareware is fake antivirus (AV) software, also known as "rogue security software." More specifically, a fake AV program impersonates an antivirus scanner and displays misleading or fraudulent alerts in an attempt to dupe a victim into purchasing a license for a commercial version that is capable of removing nonexistent security threats.

So, what did the Santa Barbara team do? Well, they:

have been able to acquire backend servers for several multi-million dollar criminal operations selling fake AV products.

...

Since we have access to the servers used by these criminal organizations, we are able to directly analyze the tools that are used to create the fake AV products, including programs that assist perpetrators in controlling the malware's behavior and brand names, as well as custom packers that obfuscate the malware to evade detection by legitimate antivirus products.

And what is it that they learned? Well, (quoting again):

the modus operandi of the criminals

the amount of money involved

the victims who purchase the software

the affiliate networks that promote the campaigns

the flow of money from the victims's credit cards, to the payment processors, to the bank accounts controlled by the criminals.

That is, basically, everything.

As they put it:

This unprecedented access allowed us to obtain ground truth about the type and sophistication of the techniques used to lure victims into paying for scareware, as well as the amount of transactions performed, including refunds and chargebacks.

One of the most chilling sections of the paper is the part where the authors explore the fuzzy, vague, blurred line between modern organized crime, and the core operations of the modern Internet:

An interesting facet of fake AV sales is the process in which credit card transactions are handled. In particular, payment processors (also known as payment service providers) are an integral part of every sale. Without these processors, fake AV operations would not be able to accept credit card payments. This would make it not only harder for a victim to purchase the product (i.e., they would have to use an alternative form of payment, such as cash, check, or money order), but it would also likely raise red flags that the software may be fraudulent. Note that payment processors must maintain a degree of legitimacy, or they risk losing the ability to accept major credit cards.

...

Perhaps the most notorious payment service provider is Chronopay, which ... has long been associated with processing transactions for various forms of online criminal organizations ... [H]owever ... also provides legitimate services to large organizations such as [ an amazing list of top-shelf names follows ]

I'm not kidding about this paper. You'll think you're reading something from Ludlum or LeCarre, but you're not. This is real life, in the modern world, on the Internet.

In the article Somers discusses his own personal history about learning how to program, and describes some of his early attempts at learning to program by reading programming books:

I imagined myself working montage-like through the book, smoothly accruing expertise one chapter at a time.

What happened instead is that I burned out after a week. The text itself was dense and unsmiling; the exercises were difficult. It was quite possibly the least fun I've ever had with a book, or, for that matter, with anything at all. I dropped it as quickly as I had picked it up.

I've read those books, plenty of them, and they are "dense and unsmiling" indeed. Mr. Somers, I feel your pain!

But then Somers got lucky, and stumbled onto an early version of Colin Hughes's wonderful website, Project Euler.

Somers goes on to describe the effect upon somebody who has an aptitude for programming, when they encounter the Project Euler problem set:

What's especially neat about it is that someone who has never programmed -- someone who doesn't even know what a program is -- can learn to write code that solves this problem in less than three hours. I've seen it happen. All it takes is a little hunger. You just have to want the answer.

It's a very insightful description, and quite accurate, I suspect. If you are a natural programmer, then looking at the problems on the Project Euler website will have an immediate effect on you. As soon as you see Problem 1, written almost a decade ago but still just as fresh now as it will be 100 years from now, you aren't going to be able to think about anything else until you go find a computer somewhere close to hand and bash it out:

Now, there's no magic here. Programming is hard work, and, like any well-developed skill, it takes practice, practice, and yet more practice (take it away, Abstruse Goose!) Puzzle-solving techniques like these can help you stay motivated, and can help you progress from the simple to the more complex, but it's up to you to persevere, and to sweat the details.

But as Somers notes, this is what learning should be like, and this is what the Web is supposed to be all about:

What you'll find there is something that educators, technologists and journalists have been talking about for decades. And for nine years it's been quietly thriving on this site. It's the global, distributed classroom, a nurturing community of self-motivated learners -- old, young, from more than two hundred countries -- all sharing in the pleasure of finding things out.

I heartily concur.

So thanks, Mom, for sending me the link, and thanks Colin Hughes for the fascinating website, and thanks James Somers for the thoughtful essay.

Saturday, June 11, 2011

A system comprising: units of a commodity that can be used by respective users in different locations, a user interface, which is part of each of the units of the commodity, configured to provide a medium for two-way local interaction between one of the users and the corresponding unit of the commodity, and further configured to elicit, from a user, information about the user's perception of the commodity, a memory within each of the units of the commodity capable of storing results of the two-way local interaction, the results including elicited information about user perception of the commodity, a communication element associated with each of the units of the commodity capable of carrying results of the two-way local interaction from each of the units of the commodity to a central location, and a component capable of managing the interactions of the users in different locations and collecting the results of the interactions at the central location.

This is the core paragraph of United States Patent 7,222,078, which was applied for by Daniel Abelow of Newton Massachusetts in December, 2003, and was granted by the Patent Office in May, 2007.

But what language is the above paragraph written in? It is not the language of software engineers: it is complete gibberish to me; it might as well be written in Urdu for all the meaning I can get out of it. After 30 years of software engineering, I can sit down in front of nearly any description of a piece of software and within a few sentences I can grasp what the author is trying to describe, and compare and contrast it to other similar descriptions that were written by software engineers, for software engineers.

This description is written in the language of lawyers, and I have no idea what they are talking about. What is a "commodity", and what are "units of a commodity"? What is a "two-way local interaction", and what sort of "memory" is it that needs to be "capable of storing results", since I know of no other sort of "memory" that is used in computer software.

Since no software engineers use language like this, there must have been a translation process:

Initially, the computer software in question was described by its engineers, in the language of software engineering, stating its design and how it worked.

Then, some person or persons must have translated this description into the language of lawyers, presumably because the United States Patent Office only accepts applications written in the language of lawyers.

Then, the legal description must have been re-translated back into the language of software engineering, so that experienced software professionals similar to myself could examine the description, understand whether it was meaningful and clear, and determine whether or not it was an invention worthy of a United States Patent.

I suppose that I am faced with two questions, one rather pragmatic, and one more philosophical:

Is it possible to locate either or both of the descriptions of a patent which are written in the language of software engineers, so that I could read, e.g., the description of a patent such as United States Patent 7,222,078 in a language I can understand?

Wouldn't the patent system work better if patents were applied for and granted in their actual language, rather than being translated into the language of lawyers? Since every language translation is fraught with ambiguities and errors, why do we force extra unnecessary translations such as these?

The One Page Dungeon Contest is a wonderful exercise that forces contributions to combine various aspects (design, story, technique) of the dungeon-maker's art into a single compact format. As with a number of creative efforts, the discipline of constraining yourself to a single page has a number of benefits, most importantly it forces you to focus and simplify and identify the essence of your idea.

As Einstein's wording of William of Occam's principle as it: everything should be as simple as possible, but no simpler.

I find that this paring-down-to-the-essence technique works wonders in software engineering, too, by the way. When you are concentrating on a bit of code, the question you should always have in your mind is not "what can I add?", but rather "what can I take out?".

But back to dungeons.

I think that the best way to enjoy the One Page Dungeon Contest is to look through the winning submissions, find one which appeals to you, and then look at the bottom of the campaign wiki where the committee has cross-referenced the supporting essays, articles, and blog posts from the authors of the dungeons.

For example, here's the nifty Citadel of Evil, by Stuart Robertson. It's gorgeous, but what's even better is Stuart's blog post, where he goes into a nice explanation of the "pocketmod" format for packing a booklet into a single foldable page with some great references to other uses of that technique.

Not much actual chess in the article though. For that, take yourself off to Romania, where the 5th edition of the Bazna King's Tournament gets going in about an hour. Sponsored by the Romanian natural gas company, RomGaz, the tournament takes place in this museum. The lineup is superb, hopefully the games will be superb as well!

Thursday, June 9, 2011

Wilson was previously the editor of Beautiful Code, a book I found very interesting and stimulating. One of the chapters in Beautiful Code is written by the founders of my day job, but that's not the only, nor even the primary, reason that I enjoyed the book. It's just quite a good book.

The Architecture of Open Source Applications at first blush looks to be relatively similar: it is a collection of contributed essays by a variety of authors; the common thread is that each author is writing about a particular Open Source project. From the overview:

the authors of twenty-five open source applications explain how their software is structured, and why. What are each program's major components? How do they interact? And what did their builders learn during their development? In answering these questions, the contributors to this book provide unique insights into how they think.

The book looks quite intriguing, with a substantial list of well-known authors contributing their thoughts on important projects. Here's the table of contents:

The goal is to preserve one copy of every published work. The universe of unique titles has been estimated at close to one hundred million items. Many of these are rare or unique, so we do not expect most of these to come to the Internet Archive; they will instead remain in their current libraries. But the opportunity to preserve over ten million items is possible, so we have designed a system that will expand to this level. Ten million books is approximately the size of a world-class university library or public library, so we see this as a worthwhile goal. If we are successful, then this set of cultural materials will last for centuries and could be beneficial in ways that we cannot predict.

Even "centuries", of course, is not very long. But it's far, far better than doing nothing at all.

Saturday, June 4, 2011

In our family data center, we've recently been upgrading from our previous file server solution, which was a 3Ware RAID card on the master Linux workstation, to a sparkling new Data Robotics (hmmm maybe now they just call themselves "Drobo"?) Drobo FS.

The box itself is gorgeous and simple. On the back, there are two connections: power, and Ethernet. On the front, the box communicates with lights. There are green, yellow, red, and blue lights, with a variety of meanings. Basically, green means the box is running and the disks are happy; blue tells you how full your system is (we're barely 10 % used, though that will quickly grow as we migrate data to the box).

Inside the box, there are five slots, each of which takes a SATA 3.5 inch hard drive. Yes, that's the entire specification of the disk drive requirements:

Accommodates from one to five 3.5” SATA I / SATA II hard drives of any manufacturer, capacity,spindle speed, and/or cache.

On your network, the Drobo makes itself available as a completely encapsulated Windows file server. Drobo provide a very simple "dashboard" management tool, with which you can connect to the device, configure user access, monitor system conditions, start/stop/restart the device, control network configuration, create and manage filesystems, etc.

By default, the machine grabs a network position using DHCP and then waits patiently for you to connect with the dashboard and start configuring shares.

Connecting to the Drobo from the 4 Windows machines was nearly instant; the Drobo is incredibly friendly to a small Windows network.

Connecting from the Linux box was a bit more challenging, though we soon worked it out. These are the essential steps (for our Ubuntu machine):

Ensure that you have the smbfs application suite installed on your Linux box to enable things like mount.cifs

Give the Drobo a static IP address on your network for ease of describing mount commands, or at least give it a DNS name that works from your other devices.

Using standard Linux commands, mount the Drobo file shares, and away you go!

I got a little bit confused about how to tell the Linux box to mount and access the device as a non-root user; my first mount attempts resulted in mounting the Drobo so that only the root account on the Linux box could access it.

With a bit of experimentation, we discovered that the mount.cifs tool appears to use some combination of the information in the mount command, and the ownership and permissions on the mountpoint, to decide how to access the remote filesystem, so we did this: