Monday, January 20, 2014

Rethinking data centers, one node at a time

Bittorrent and bitcoin have proven the power of peer-to-peer, decentralized networking. Amazon Web Services and Rackspace Cloud demonstrate the market desire for cloud storage and cloud processing. Why not combine these into a single package?

The storage system Tahoe LAFS demonstrates how the data center may be inverted, making the end user the ultimate provider of data storage services. Continue this idea writ large, and apply it to cloud storage, cloud processing, and other cloud services.

Now, boil cloud services down to their primary components, and consider a user interface that anyone, even largely computer illiterate, could manage.

blackbox: core components

Ultimately, computers provide resources. These resources may be enumerated, and resold on the free market. For cloud services, these resources are

Given these few metrics, we can conceive of "blackbox" software that connects to TradeNet, offers the above resources on the free market, and receives payment when the resources are utilized.

The user experience is as simple as it gets: Turn on your computer, run the blackbox, and get paid.

Automated service, cost routing

blackbox is not a new idea. These sorts of ideas are decades old. The new twist is adding an efficient digital currency (bitcoin) with online, automated, decentralized markets.

Computing jobs, be it 24/7 real-time services ("run my website") or periodic batch processing ("balance today's accounting and sales records") will be layered on top of blackbox's low level cloud services in precisely the same manner as cloud services are used today.

Based on your service needs, nodes will be selected based on cost, reliability, reputation, geolocation, speed, and many other factors. Requests for service will be matched with offers for service. Contracts will be agreed upon, perhaps lasting only 24 hours, or an hour, or 60 seconds. Payment terms may be guaranteed through multi-signature transactions, with reputation, escrow and arbitration ensuring that all parties have incentives to deliver.

Keep us online and honest

The two primary weak points of a decentralized, generalized cloud network are resource accounting and network defense.

Resource accounting includes issues such as ensuring that participants X, Y, and Z all deliver their contractually-promised resources, or are highlighted for lack of delivery. Service delivery fails for any number of reasons: malicious action, hardware or software failure, human error. Ultimately the reasons for service delivery failure are not important. Nodes will develop a reputation for good or bad service over time.

Network defense will involve associating and disassociating with various parties based on their reputation. As in Tahoe-LAFS, most consumers of computing services will communicate with layers of software 1-2 layers removed from the low-level blackbox software itself. This offers privacy for the provider, as well as shielding blackbox users themselves from many attacks. Other defenses will depend more on the precise P2P technologies involved in TradeNet and blackbox.

Mesh network writ large

Mesh networks provide an example of a large, useful service collectively built from tiny components. TradeNet takes the concept further, imagining a mesh of markets. blackbox is simply the next logical extension of thought: once easy, automated, anyone-can-join markets exist, it becomes easy to resell one's own computing resources using modern software that combines those computing resources into a greater collective whole.

Once these P2P cloud services are available, developers will create more useful software layered on top: web hosting, email services, video processing, all the services one can perform in the cloud today. Presumably, with the free market at work, costs would be lowered for many, while utilizing computing power that would otherwise be sitting around idle.

With blackbox, a user enters their bitcoin address, and tells the software to make use of the computer, when the owner is away. Computers inside data centers will run blackbox 24/7.

blackbox 2.0

All of the above may be implemented, in theory, given today's technology. Improving the system further involves additional data, permitting additional price discovery.

Recent hardware trends have included a focus on efficiency and power usage, a welcome change from decades of counting CPU cycles/second. blackbox 2.0 would examine local electricity prices, and determine if contributing a resource is economically viable, considering the power usage involved.