All of us developers can write code. We need designers for the look,feel, page designs, and flows. To get the services of a web designer, often we need to do a prototype. If the design is bad, nobody may even see the potential of the website. For a lot of web apps, there may not even be enough budget to attract a designer.

What should the developers do? Can they do the web design by themselves? Even if they can’t do a world-class design, can they create reasonably good web pages? This note, created from some of the lessons I taught my developers, answers the questions.

Small rant: Historically, clothes were hand made. They were ill-fitting, expensive, and of poor quality. After Mr. Singer came up with machine, they were cheaper and had better quality. Over the time, they became reasonably well-fitted too. Still, a modern custom Italian suit is better than a mass produced suit.

Most people hiring UX designers want that Italian design better than the machine design. But, in reality, they are getting the ill-fitting, expensive medieval designs. For them, a better choice is a machine design – that is, a factory approach, using standards based designs, with mass customization. It is an engineering approach like that of Mr. Issac Merrit Singer’s. If you have the right budget, of course, you can go for high end tailors or high end designers.

If you have not read them already, please read the following blog posts:

If you are strapped for time, you can skip to the last section, where I give you a standard way you can develop web pages, with good chance of success.

We’re not going for the originality. We are looking to use standard resources, standard tools, and standard best practices. let us study the core elements of a web design: Styles and Trends, UI Elements, Interactions, Fonts, Icons, Colors, and Layouts.

Styles and Trends

This is my soon-to-be-outdated advice: follow Google style. Use affordance where you need instead of full flat design. Don’t go for Skeumorphic designs, as they are difficult to design and maintain.

Skeuomorphism: Imitating the real world. That is, when you find a notebook imitating physical notebook, that is skeuomorphic design.

There are positive points for this design, of course. The user instantly recognizes it by identifying with the real-world counterpart. For novice users, it is a worthwhile association. For example, if a novice user sees a notebook with familiar ruled yellow paper, they know what it is for.

But, the flip side is that it is difficult to design. And, it gets annoying quickly. For an experienced user, it is hurdle for frequent usage.

Follow the link to see the set of examples to understand how it can easily go overboard.

Even Apple realized the excesses of Skeumorphic design and began to prefer flat design. This design reduces clutter and simplifies the appearance. However, flat design loses affordance. That is, if there is a button, you press it. If there is patch of color, you don’t think of pressing. Still, it is the recent trend. You can find different themes for the standard layouts. I find it quite useful to present information.

For instance, the above picture is good illustration of information presentation using flat design. You can see lot of examples of the flat widgets in Flat-UI project.

How about bringing some affordances to flat design? See http://sachagreif.com/flat-pixels/ for ideas on how to do it. Simply put, you can do flat design, but add shadows, gradients, and even buttons with relief to retain the usability of the design.

UI Elements

A page is composed of several UI elements. Some are basic, a standard part of HTML, such as tables, buttons, forms etc. Some are built with these basic elements: menus, breadcrumbs etc. Here are some simple rules in creating a standard set of UI elements:

Do not design any UI elements. Borrow from a consistent set. Please see the layouts section for further details on standard set of UI elements that come with toolkits like bootstrap.

For higher-order UI elements, again, do not design your own ones. If the toolkit provides them, use them. If you must design, create a catalog of semantic elements you need and use that to guide a standard set of elements.

http://www.cssbake.com/ – more focus on the basic elements – these can be used to spruce up the ones that come with the layout toolkit.

Interactions

These days, the web page design is not static. Depending on the user interactions, we need to change the page. For instance, when the user selects a particular part of the page to interact, perhaps it makes sense to remove the unneeded parts of the page. Lots of these interactions are accomplished with JQuery and its plugins. Some of the standard interactions are table design and infinite scrolling that you see on Facebook.

To use the fonts well, you need to understand how to use sizing of fonts to your advantage. If you use the UI patterns appropriately, you will know how to use the right font size to do a call out, or promote. You also will understand how to use colors to indicate the information classification to the users.

Icons

Icons make information easily identifiable and usable. Even the simplest icons provide information quickly. For example, take Otl Aicher’s stick figures: he designed the icons for the Munich Olympics and changed the way public communication occurs through pictograms.

In web, icons play even bigger role. There are two ways to use icons:

Using icons as images: For instance, you can find many sets of icons that are free to use in your website. All you need to incorporate these icons is to download them and use their jpg/gif/svg in your website.

Using icons as font: The problem with using icon images is that you cannot manipulate them. For instance, you cannot resize them (in svg, you can, but in others, they lose fidelity). You cannot change the color (you need multiple sets). You cannot transform them. Visit https://css-tricks.com/examples/IconFont/ to understand how icon fonts can be colored, resized, slanted, shadowed etc.
If you are using icon fonts, you can start with: http://fortawesome.github.io/Font-Awesome/ that go well with bootstrap. Or, you could look at comprehensive library like: http://weloveiconfonts.com/

Still, if you need icons with multi-colors, you need to use the images.

Colors

Lot of engineers feel challenged when asked to choose the colors. They resort to bland colors that do not work well with each other. If you are choosing a theme designed by an in-house designer, or a theme provider, the choices would be made for you. If you need to customize a standard theme, you can consider the following:

https://kuler.adobe.com/create/color-wheel/ – the color wheel is a standard way of choosing a set of colors that go well together. There are different variations – monochromatic to triad, or complementary to analogous. Choose one color and play with the combinations that work well with that color.

http://colorco.de is also a nice interface to using color wheel. Feel free to select the type of the color scheme you want and move the cursor over the colors to vary the combinations.

Layouts

The layout of the elements of the page is a crucial step in web design. The earliest designs did not understand the web as a different medium than desktop applications and gave us the designs laden with the same kind of menu’s with dropdown choices. Web is a dynamic medium that can adjust based on the context, the user, and the situation.

There are two parts to the layout: what should be in the layout, and how they should be laid out.

Conceptual elements in a layout

What should be in each page or layout is a much bigger topic than this simple post. I will describe in a separate post. Meanwhile, here are the fundamental rules to remember:

Make the pages task oriented for most post. If we need exploratory elements, use them as right recommendations in the page.

Do not clutter the page with all possible choices. Give the choices that make sense only in that context.

Give the user ability to escape the current thread of flow.

Feel free to hide and show the elements on demand. That is, we do not need to get a new page for every change in the page.

Respect the URL – The user should be able to bookmark and land on a page and carry on transaction from there; or, the user can do back and forth among the URLs.

Set a few standard page elements that reduces the need to learn for the users.

Soon, in other posts, I will describe the process of designing the elements of a layout.

Physical layout

Physical layout part has become simpler. These days, if somebody is developing the front end, the layout should satisfy the following:

It should work on any (modern) browser: Browser wars are oh so ancient. The web page should be viewable on any browser. However, since most modern web technologies require modern browsers, we can assume usage of modern browser (thanks to the mobile revolution), that is beyond IE7. Companies like Google already are pushing the market beyond even IE9. Browsers like Chrome and Firefox keep themselves updated to support most modern features.

It should work on any form factor: The layout should support any size of the browser. Some devices can only support smaller size browser; some support different orientations; some support different resolutions. Our layout should work on all these varying sizes, orientations, and resolutions.

We can check for the browser size (and the device, while we are at it), and generate the appropriate HTML. This approach doesn’t work well with proliferation of variations in sizes. Besides, what if we resize (or reorient) the browser after we got the page? Reloading the page is so retrograde, and breaks user experience (think re-posting a purchase – not what user expects).

We can indicate using CSS on how to layout: that is, no absolute sizes – only relative metrics. Using the right kind of weights, CSS positioning, we may be able to achieve the design we want.

We can add JS that can redraw the page based on the size: By adding JS that can hide or show elements, we can enhance the CSS to support devices even better. For instance, why show a full side bar with a menu, when we are seeing it on a mobile device, where there is barely enough space to display the main content?

While those are typical choices, in practice, you will use one of the following frameworks. These frameworks incorporate CSS and JS to deliver responsive design:

Bootstrap: The most popular choice for responsive design. You can customize what you need and get only the bare minimum needed. As a bonus, you will get a fully integrated set of icons, widgets, JQuery and plugins, and ability to customize the L&F of the site.

Zurb Foundation: Very similar to Bootstrap. The approach is more of a toolkit – it lets you design what you want. It has limited set of UI elements, and is not as opinionated as Bootstrap is.

Pure css: If you cannot use JS (because of organizational policies of using Javascript), you can always use Pure which is a pure css based responsive layout.

There are several other layout frameworks like skeleton with minor variations on these categories. The popular ones like bootstrap come with standard themes. These themes add standard layouts, colors, fonts, images, icons, and even templates. For example:

Every once in a while, I get the urge to work with computers. I want to get my hands dirty, figuratively, and dig into the details of installation, configuration, and execution. This experimentation comes in handy when we discuss the trends in the enterprise. Typically, we neglect processes when we do small scale experiments, but that is matter for another time. Besides, it is really fun to play with new technologies and understand the direction these technologies are heading to.

My personal machine

I wanted to run virtual machines on my system, instead of messing with my own machine. Because I multiplex a lot, I want to have large enough server. That way, I can keep all the VM’s open instead of waiting for the VM’s to come up, when I need them.

Since my basic requirement is to have large amount of memory, I settled on Sabertooth X79 mobo. It can support 64GB, which is good enough to run at least 8 VM’s simultaneously. Someday, I can convert it to my private cloud instance, but till then, I can use it as my desktop machine with lot of personal VM’s running.

I have two 27” monitors ordered off ebay, directly from Korea. Each monitor, costing $320, offers 2560×1440 resolution, with stunning IPS display – it is the same as in Samsung Galaxy, but with large 27” diagonal size. These days, you can get them from even newegg.

To support these monitors, you need dual DVI – two of them. They do not support HDMI and VGA would negate all the benefits of such high resolution. The consumer grade reasonable one is built with GeForce GT 640, of which there are several.

Finally, I used pcpartpicker site (http://pcpartpicker.com/p/23MXv ) to put together all my parts and it showed if my build is compatible internally or not. Also, it helped me pick the stores where I can buy them from. I ended up ordering from newegg and Amazon, for most needs. I also had all other needed peripherals like Logitech mouse, webcam, and MS Keyboard etc. from before, which I used for my new computer.

Software

For software, I opted to use Windows 8.1, as I use office apps most of the time. I use ninite.com to install all my apps – they can install all the needed free apps. Here are some of the apps I installed using that app: Chrome, Firefox, VLC, Java, Windirstat, Glary, Classic Start, Python, Filezilla, Putty, Eclipse, Dropbox, Google Drive.

Since I needed to run VM’s on this machine, I had a choice of VMPlayer or Virtual Box. I opted for VMPlayer.

My cloud machine

While the personal machine is interesting, that was only to free up my existing 32GB machine. The cost of such a machine, with right components is less than $1000. As per software, I had the choice of using ESXi 5.5, Xenserver 6.2, or Microsoft hypervisor 2012 R2. All of them are free, which meets my budget.

I tried ESXi (VMWare VSphere hypervisor), which did not recognize my NIC on my mother board. I tried inserting the driver in the ISO from previous release, but even after recognizing the Realtek 8111 nic, it still did not work. Xensever, on the other hand, worked perfectly well with first try. Since yesterday, I have been playing with Hadoop based Linux versions in this setup.

If you want to try

It is fairly cheap to have your own private setup to experiment. Here is what you can do:

Get yourself a decent quad-core machine with 32 GB. You do not need dvd drive etc. Add couple of 3TB disks (the best is Seagate Barracuda, for the right price). If you can, get a separate NIC (Intel Pro 1000 is preferred, as it is best supported).

Install Xenserver on the machine. It is nothing but a custom version of Linux, with Xen virtualization. You can login like any Linux machine as well. The basic interface, though, is a curses based interface to manage the network and other resources. [Image courtesy: http://www.vmguru.nl/ – mine was 6.2 version and looks the same. From 6.2 version, it is fully open source.]

On your laptop, install Xencenter, which is the client machine for it. Xencenter is full-fledged client, with lot whizbangs. It has support to get to console for the machine and other monitoring help. We can use the center to install machines (from a local ISO repo), convert from VMDK to OVF format for importing etc.

It is best to create machines for it, as conversion is a little error prone. I created a custom Centos 6.4, 64bit machine. I used it as my minimal install.

When I installed it, the installation did not allow me to choose a full install. That is, it ended up installing only basic packages. I did the following to get a full install:

The console doesn’t seem to offer the right support for X. So, I wanted to have VNCserver with client running on my Windows box.

I installed all the needed RPM directly from the CD’s, using the following commands:

I added the CDROM as a device for the YUM repo. All I needed were a few edits in the yum.repos.d folder.

I mounted the CDROM on Linux (“mount /dev/xvdd /media/cdrom” : notice that the cdrom device is available as /dev/xvdd).

I installed VNC server and customized to open for my display size of 2260×1440.

In the end, I removed the peripherals, and made the server headless, and stuck it in the closet. With wake-on-lan configured, I never need to visit the server physically.

At the end, you will have a standard machine to play with, a set of minimal installs for me to experiment with on your XenCenter.

What you can do with it

Now, you do not have a full private data center. For instance, you don’t have machines to migrate the VM’s to, setup complex networking among the machines, and connect storage to compute servers. For even with this, you can do the following activities:

Setup a sample Hadoop cluster to experiment: It is easy enough to start with Apache Hadoop distribution itself so that you can understand the nitty gritty details. There are simple tasks to test out the Hadoop clusters.

Setup a performance test center for different NoSQL databases. And, do the performance tests. Of course, performance measurements under VM’s cannot be trusted as valid, but at least you will gain expertise in the area.

There is a lot of interest in moving applications to the cloud. Considering that there is no unanimous definition of cloud, most people do not understand the right approach to migrate to the cloud. In addition, the concept of migration itself is complex; what constitutes an application is also not easy to define.

There are different ways to interpret cloud. You could have private or public cloud. You could have just data center for hire or a full-fledged, highly stylized platform. You could have managed servers or instead measure in terms of computing units, without seeing any servers.

As we move applications to any of these different kinds of clouds, you will see different choices in the way we move the applications.

Moving a simple application

Let us consider a simple application.

The application is straightforward. Two or three machines run different pieces of software and produce a web-based experience to the customers. Now, how does this application translate to the cloud?

As-is to as is moving

Technically, we can move the machines as-is to a new data center, which is what most people do with the cloud. The notable points are:

To move to “cloud” (in this case, just another data center), we may have to virtualize the individual servers. Yes, we can potentially run whatever OS on whatever hardware, but most cloud companies do not agree. So, you are stuck with X64 and possibly, Linux, Windows, and a few other X64 OS’s (FreeBSD, illumos, smartOS and also variants of Linux).

To move to cloud, we may need to setup the network appropriately. Only the web server needs to be exposed, unlike the other two servers. Additionally, all three machines should be in LAN for high bandwidth communication.

While all the machines may have to be virtualized, database machine is something special. Several data bases, Oracle included, do not support virtualization. Sure, they will run fine in VM’s, but the performance may suffer a bit.

In addition, databases have built-in virtualization. They support multiple users, multiple databases, with (limited) guarantees of individual performances. A cloud provider may offer “database as a service” which we are not using now.

In summary, we can move applications as-is to as-is, but we still may have to move to X64 platform. Other than that, there are no major risks associated with this move. The big question is, “what are the benefits of such a move?” The answer is not always clear. It could be a strategic move; it could be justified by the collective move of several other apps. Or, it could be the right time before making the investment commitment to the data center.

Unfortunately, moving applications is not as easy. Consider the slightly more complex version of the same application:

Let us say that we are only moving the systems within the dotted lines. How do we do it? We will discuss those complexities later, once we understand how we can enhance the moving that treats cloud like a true cloud.

Migration to use the cloud services

Most cloud providers offer many services beyond infrastructure. Many of these services can be used without regard to the application itself. By incorporating into the processes and also adding new processes to support the cloud can improve the business case to moving to the cloud. For instance, these services include:

Changes to these processes and tooling is not specific to one application. However, without changing these processes and ways of working, the cloud will remain yet another data center for the IT.

Migration to support auto scaling, monitoring

If we go one step ahead, by adjusting the non-functional aspects of the applications, we can get more out of the cloud. The advantage of the cloud is the ability to handle the elasticity of the demand. In addition, paying for only what we need is very attractive for most businesses. It is a welcome relief for architects who are asked to capacity planning based on dubious business plans. It is even bigger relief to infrastructure planners who chafe at the vague capacity requirements from architects. It is much bigger relief for the finance people who need to shell out for fudge factor built into capacity by the infrastructure architects.

But, all of that can work well only if we make some adjustments in the application architecture, specifically the deployment architecture.

How does scaling happen? In vertical scaling, just move to bigger machine. The problem with this approach is the cost of the machine goes up dramatically as we scale up. Moreover, there is a natural limit to the size of the machine. If you want to have disaster recovery, you need to add one more of the same size. And, with upgrades, failures, and other kind of events, large machines do not work out economically.

Historically, that was not the case. Architects preferred scaling up as it was the easiest option. Investments into hardware went towards scaling up the machines. Still, with new internet companies, they could not scale vertically; the machines weren’t big enough. Once they figured out how to scale horizontally, why not use the most cost effective machines? Besides, a system might require the right combination of storage, memory, and compute capacity. With big machines, it wasn’t possible to tailor to the exact specs.

Thanks to VMs, we could tailor the machine capacity to the exact specs. And with cheaper machines, we could create the right kind of horizontal scaling.

However, horizontal scaling is not so easy to achieve. Suppose you are doing a large computation – say, factorization of large number. How do you do it on multiple machines? Or, if you are searching for an optimal path though all the fifty state capitals? Not so easy.

Still, several problems are easy to scale horizontally. For instance, if you are searching for records through large set of files, you could do the searching on multiple machines. Or, if you are serving web pages, different users can be served from different machines.

Considering that most applications are web based apps, they should be easy to scale. In the beginning, scaling was easy. None of the machines shared any state – that is, there is no communication among the machines was required. However, once J2EE marketing machine moved in, these application servers ended up sharing state. There are other benefits, of course. For instance, if a machine goes down, the user can be seamlessly served out of another machine.

Suppose you introduce a machine or take out a machine. The system should be adjusted so that session replication can continue to happen. What if we run one thousand machines? Would the communication work well enough? In theory it all works, but in practice it is not worth the trouble.

Scaling to large number of regular machines works well with stateless protocols, which are quite popular with the web world. If any existing system does not support this kind of architecture, it is not difficult to adjust to this architecture without wholesale surgery on the application.

Most data centers do monitoring well enough. However, in cloud, monitoring is geared towards maintenance of large number of servers; there is a greater automation built in; there is lot more log file driven automation. Most cloud operators provide their own monitoring tools instead of implementing the customer’s choice of monitoring tools. In most cases, by integrating into their tools (for instance, log file integration, events integration), we can reduce the operational costs of the cloud.

Migration to support cloud services

If you have done all that I told you to – virtualize, move to cloud, use auto-scaling, use the monitoring, what is left to implement? Plenty, as it turns out.

Most cloud providers provide lot of common services. Typically, these services operate better on scale. And, they also implement well-defined protocols or needs. For instance, AWS (Amazon Web Services) offers the following:

Given this many services, if we just go from machines to machines, we might just use EC2 and EBS. Using these services not only saves money and time, but eventually, ability to use trained engineers and third party tools.

Re-architecting a system using these services is a tough task. In my experience, the following order provides the best bang for the buck.

The actual process of taking an existing application and moving it to this kind of infrastructure is something that we will address in another article.

Re-architecting for the cloud

While there may not be the right justification for re-architecting the applications entirely, for some kind of applications, it makes sense to use the platform that the cloud providers offer. For instance, Google compute offers a seductive platform that offers the right kind of application development. Take a case of providing API for product information that your partners are embedding on their site. Since you do not know what kind of promotions your partners are running, you have no way of even guessing how much the traffic is going to be. In fact, you may need to scale really quickly.

If you are using say, Google app engine, you won’t even be aware of the machines or databases. You would use an appengine, and the APIs for big table. Or, if you are using any platforms provided by the vendors (Facebook, SFDC, etc.), you will not think of machines. Your costs will truly scale up or down without actively planning for it.

However, these platforms are suitable for only a certain kind of application patterns. For instance, if you are developing a heavy duty data transformation, a standard appengine is not appropriate.

Creating an application for a specific cloud or platform would require designing the application to make use the platform from the cloud. By also providing standard language runtime, libraries, services, the platform can lower the cost of development, I will describe the standard cloud based architectures and application patterns some other day.

Complexities in moving

Most of the complexities come from the boundaries of applications. You saw how many different ways the application can be migrated if self-defined. Now, what if there are lot of dependencies? Or, communication between applications?

Moving in groups

All things being equal, it is best to move all the applications at once. Yet, for various reasons we move only few apps at a time.

If we are migrating applications in groups, we have to worry about the density of dependencies, the communications among the applications. Broadly speaking, communication between apps can happen the following ways.

Exchange of data via files

Many applications operate on import and export (and transformation jobs in between). Even when we move a set of applications, it is easy enough to do these file based communication. Since file-based communication is typically asynchronous, it is easy to setup for the cloud.

Exchange of data via TCP/IP based protocols

In some cases, applications may be communicating via standard network protocols. Two applications may be communicating via XML over HTTP. Or, they could be communicating over standard TCP/IP with other kinds of protocols. X windows applications communicate over TCP/IP with X server. Applications can use old RPC protocols. While these protocols are not common anymore, we might encounter these kind of communications among applications.

To allow the communication to continue, we need to setup the firewall to allow such communications. Since we know the IP numbers of end points, specific ports, and specific protocols, we may be able to setup effective firewall rules to allow such communication. Or we can set up VPN between the two different locations.

It is easy to handle the network throughput; in most applications, the throughput requirements are not very high. However, it is very common to have a low latency requirement between applications. In such cases, we can consider dedicated network connection between the on-premise center and the cloud data center. In several ways, it is similar to handling multi-location data centers.

Even with a dedicated line setup, we may not be fully out of woods yet. We may still need to reduce the latency further. In some cases, we can deal with it by caching and other similar techniques. Or, better yet, we can migrate to modern integration patterns such as SOA or message bus using middleware.

Exchange of data via messages or middleware

If we are using middle ware to communicate, the problem becomes simpler. Sure, we still need to communicate between the apps, but all the communications go through the middleware. Moreover, middleware vendors are dealing with integrating applications across continents, across different data centers, and across companies.

ESB or any other variants of middleware can handle a lot of integration-related complexities. They can do transformation, caching, store and forward, and security. In fact, some of the modern integration systems are specifically targeted towards integrating with the cloud, or running integration systems in the cloud. Most cloud providers offer their own messaging systems that work not only within their clouds, but also across the Internet.

Database based communication

Now, what if applications communicate via database? For instance, an order processing system and an e-commerce system communicating via database. And, if e-commerce system is on the cloud, how does it communicate with on-the-premise system?

DB-to-DB sync has several special tools, since this is a common problem. If the application doesn’t require a real-time integration, it is easy to sync the databases. Real-time or near-real-time integration between databases requires special (and often expensive) tools. A better way is to handle the issue at the application level. That means we should plan for asynchronous integration.

Conclusion

Moving applications to cloud opens up many choices, each choice with its own costs and benefits. If we do not understand the choices and treat every kind of move as equal, we risk not getting the right kind of ROI from moving to cloud. In another post, we will discuss the cloud migration framework and how to create the business case, and also how to understand what application should migrate to which cloud and to what target state.

I attended Strata last week (Feb 11-13) in Santa Clara, CA, a big data conference. Over the years, it has become big. This year, it can be said to become mainstream — there are lot of novices around. I wanted to note my impressions for those who would have liked to attend the conference.

Exhibitors details

Big picture view

Most of the companies, alas, are not used to the enterprise world. They are from the valley, not the from the plains where much of these technologies can be used profitably. Even in innovation, there are only a few participants. Most of the energies are going in minute increments of usability of technology. Only a few companies are addressing the challenge of bringing Big Data to main stream companies that already invested in plethora of data technologies.

The established players like Teradata, Greenplum would like you to see big data as a standard way of operating along with their technologies. They position big data as relevant in places, and they provide mechanisms to use big data in conjunction with their technologies. They build connectors; they provide seamless access to big data from their own ecosystem.

[From Teradata website.]

As you can see, Teradata’s world center is solidly its existing database product(s).

The new comers like Cloudera would like to upend the equation. They compare the data warehouse with a big DSLR camera and the big data as a Smartphone. Which gets used more? While data warehouse is perfect for some uses, it is costly, cumbersome, and doesn’t get used for most places. Instead, big data is easy, with lot of advances in the pipeline, to make it easier to use. Their view is this:

[From Cloudera presentation at Strata 2014].

Historically, in place of EDH, all you had was some sort of staging area for ETL or ELT kind of work. Now, they want to enhance it to include lot more “modern” analytics, exploratory analytics, and learning systems.

These are fundamentally different views: While both see big data systems co-existing with data warehouse, the new companies see them taking on increasing role to provide ETL, analytics, and other services. The old players see it as an augmentation to the warehouse when unstructured or large data volumes are present.

As an aside, at least Cloudera presented their vision clearly. Teradata on the other hand, came in with marketese which does not offer any information on their perspective. I had to glean through several pages to understand their positioning.

A big disappointment is Pivotal. They ceded the leadership in these matters to other companies. Considering their leadership in Java, I expected them to extend Map Reduce to multiple places. That job is taken up by Berkeley folks with Spark and other tools. With lead in Greenplum HD, I thought they would define the next generation data warehouse. They have a concept called data lake, which is merely a concept. None of the people in the booth were articulate about what it is, how it can be constructed, what way it is different, and why it is interesting.

Big data analytics and learning systems

Historically, analytics field is dominated with descriptive analytics. The initial phase of predictive analytics was focusing on getting the right kind of data (for instance, TIBCO was harping on real-time information to predict events quickly). Now that we got Big data, it is not so much as getting the right data, but computing it fast. And, not just computing fast, but having the right statistical models to evaluate correlations, causations and other statistical stuff.

[From Wikipedia on Bigdata]

These topics are very difficult for most computer programmers to grasp. Just as we needed understanding of algorithms to program in the beginning, we need the knowledge of these techniques to analyze big data these days. Just as the libraries that codified the algorithms made them accessible to any programmer (think when you had to program the data structure for an associate array), new crop of companies are creating systems to make the analytics accessible to programmers.

SQL in many bottles

A big problem with most big data systems is the not having relational structure. Big data proponents may rile against the confines of relational structures, but they are not going to fight against SQL systems. Lot of third party systems assume SQL like capabilities from the backend systems. And, lot of people are familiar with SQL systems. SQL is remarkably succinct and expressive for several natural activities on Data.

A distinct trend is to slap on SQL interface onto non-SQL data. For example presto does SQL on Big data. Or, impala does SQL on Hadoop. Pivotal does Hawq. Hortonworks does Stinger. Several of them modify SQL slightly to make it work with reasonable semantics.

Visualization

Big data conference is big on visualization. The key insight is that visualization is not something that enhances analytics or insights. It itself is a facet of analytics; it itself is an insight. Proper visualization is the key to so many other initiatives:

Design time tools for various activities, including data transformation.

Monitoring tools on the web

Analytics visualization

Interactive and exploratory analytics

The big story is D3.js. How a purely technical library like D3.js has become the de facto visualization library is something that we will revisit some other day.

Summary

I am disappointed with the state of big data. Lot of companies are chasing the technology end of the big data, with minute segmentation. The real challenges are adoption in the enterprises, where the endless details of big data and too many choices increase the complexity of solutions. These companies are not able to tell businesses why and how they should use Big data. Instead, they collude with analysts, media, and a few well-publicized cases to drum up hype.

Still, Big data is real. It will grow up. It will reduce the costs of data so dramatically to support new ways of doing old things. And, with right confluence of statistics and machine learning, we will see the fruits of big data in every industry. That is, doing new things in entirely in new ways.

The usability of banking took a big leap, with the invention of branch office. In modern times, in the 12th century, the Templars created a network of branches, taking the banking business to where there is a need – for instance to middle east and England. They allowed the movement of funds, currency conversion, and ransom payments to happen smoothly in those days.

In recent times, a prominent feature of wild west town is an imposing bank building. This branch office provided a life line of credit to the local citizens. Along with rail road, post office, church, school, and news paper, bank branch provided the underpinnings of civilization. The building is meant to convey stability, a much needed quality, as fly-by-night operators running off with depositor’s money were common in those days. Bank of America is supposed to have spearheaded the growth of satellite branches in the US.

In the 20th century, with the advent of telephone, traditional banking got extended slightly. Unlike before, you do not need to go to the bank to carry on certain kind of transactions. You could call and enquire about the status of a transaction. You could call and even initiate certain transactions. Still, if you needed cash, you needed to go to the bank.

Credit cards changed the situation quite drastically, starting in the 60’s. You could charge the card for various purchases. In a way, it created an alternative to traditional cash. You are not using the cash, but the credit letter that the bank gave you in place of cash, in a fashion.

ATM’s changed even that situation. You can get cash when you need it – in a mall, in a super market, at a train station, and even in a casino. It truly untethered us from the branch office.

Internet: How it changed the banking

Considering that we have a way of carrying on transactions without even a branch office, do we really need branch office? That would the natural question that we may ask, looking at the trends.

As soon as internet became reliable, traditional banks have taken a different approach. They did not see it as a replacement to the existing channels, but yet another channel to serve customers. They created websites, exposing the online transactional systems and querying systems to the consumers. As technology, and adoption of technology improved, they improved the websites. They added even mobile apps.

Gamification to get users to behave in certain ways – for instance, improving saving rates, planning properly and so on.

Adding new products such as group banking etc.

In most situations, banks see these efforts augmenting their traditional channels. In fact, the biggest effort these days is to reconcile these different channels. Integration of data (for example, getting the same amount of balance on iPhone app or ATM), integration of processes (for example, starting a wire transfer online and finishing at the branch) are some of the challenges in this channel unification effort.

Modern banks have taken a different route. Since they have not established branch offices, they bypass that infrastructure, and make it a virtue. They offer better interest rates, better usability of the applications, and better customer service. For example, check out http://www.bankrate.com/compare-rates.aspx to see the best rates – they are offered by banks with no local branches. Bank Simple, which tries to offer superior technology service, has gained more $1B deposits within an year of opening, without any track record.

[Simple.com’s mobile application].

Surprisingly, a bank’s ability to attract customers is directly proportional to the number of branch offices they have in the neighborhood. [See: http://www.economist.com/node/21554746]. However, with the changing demographics, wider adoption of technology, and the pressure from different industries, the situation is changing.

Web 2.0: How it will change the banking

Whether banks view internet applications as an another channel, or the primary channel, the focus has been always about improving their applications: websites, mobile applications, internal applications. Yet, the biggest financial innovation of the early internet, PayPal, does none of that.

Technology wise speaking, PayPal succeeded in taking the ball where the game is, instead of insisting people come to its playground. It successfully integrated into several online store fronts. It is almost like it setup ATM’s all over the internet, at the moment of purchase.

When we look at other industries, we see the same trend. Instead of assuming the burden of developing the applications consumers want, they allow others to develop apps. With extreme segmentation, they allow multiple groups to develop and serve different segments as those groups seek to serve. In fact, several companies use API’s are a way to increase awareness with internal departments, external partners, and potential employees. They embrace it to such an extent, that they even hold hackathons to create apps.

In mid 90’s, I read a paper called, “It’s bits, stupid”, a take-off on Clinton’s “It’s economy, stupid”. The concept is that the telephone companies controlled the telephone applications from beginning to end. Want to introduce three way calling? You need to go and change the switch code, change the telephone handsets etc. Want to have call hunting? Again, you need to change code in the switch etc.

Compare it with internet, where it was only interested in pushing bits. Building the actual apps was left to the ecosystem. Internet, web, VOIP, Google hangouts – all these were result of that innovation. To think that SS7 could have been TCP/IP or even could have assumed the same openness as TCP/IP is unimaginable these days.

In fact, even in the staid old world of telephony, one of the most successful companies in creating an ecosystem is twilio. Using its API’s people have crafted different applications ranging for customer service apps, SMS apps, and helpdesk apps.

[Twilio has the ability to analysis of the calls – this app is put together on top of Twilio API’s. Copyright: Twilio.]

If Banks have to embrace this way of participating in a large ecosystem, they need to change the way they develop applications. They could take cues from successful companies like Twitter and Facebook. Twitter built its entire business through API’s allowing users to share stories, comment from within the applications. So did Facebook. Let us see how companies are embracing this philosophy of separation of core API’s and apps, .

API economy

When we look at companies that are successful at fostering an ecosystem where others can participate in developing applications, we find the following:

They make it easy for others to use the API’s.

The standard, routine, or the core portion of the logic is managed by the company. The customization, specialization etc. are delegated to the ecosystem.

They allow the users to integrate into their workflows and ways of working.

Even if the companies are not interested in exposing the APIs to general public, they are interested in going this route at least for internal audience. For one thing, in several large companies, different groups behave as perfect strangers – therefore, all the standard techniques of getting developers to adapt your platform and API’s apply here. For another, the technical and engineering advantages are increasingly in favor of this approach.

Banking trends

For banks, the API’s offer an interesting mix of privacy, convenience, security and trust. For instance, PayPal offers privacy (they need not know my cc number), trust (they can trust that PayPal will pay out and do any dispute management). The most popular with new web companies, stripe, offers both, without the burden of keeping track of payments, or regulatory compliance of keeping the CC numbers.

The tug-of-war we see these days is between these two: trust and privacy. Lot of people hate PayPal because they do not trust its track record as the arbitrator. That is, it is protecting privacy, even at the expense of trust. Cash for example, offers a good balance between trust and privacy. However, it is not convenient. Bitcoin offers perfect anonymity, and little less of trust. Banks offer great deal of trust, but little less anonymity.

[Does popularity = trust? At least in Bitcoin case, it seems to be so.]

The current generation is losing its trust in governments. With the rise of citizen journalism, governments are seen as cynic at best, or corrupt at worst. Banks, aligned to government through fiscal policies, are tainted by the same guilt. While the current business does not suffer, and even the future business – commercial and high net-worth business may not suffer, individuals may eventually find alternatives to banking.

Hopefully, with the right API’s banks will relinquish some of the power they hold, for which they are blamed. If all I am doing is facilitating a payment, then, I cannot be held responsible for the application built on it, correct? While the laws catch up to the creative frenzy of the internet, banks will end up focusing on providing safe, proven, trusted, and secure services.

Incidentally, banks already offer API’s, whether in proper technical form or not. They work with tax prep software to get the tax details. They work with aggregators like mint.com, sigfig.com, yodlee.com for the get the details of the user accounts for analytic purpose. Most of these aggregators built solutions to get the account details from banks, but lot of those solutions are brittle, without support from banks.

[Mint.com got the information from two accounts here: Etrade, Fidelity and showing the analysis].

Technical trends

Loosely speaking, APIs are SOA for the easy app development. Most modern API’s are simply JSON over HTTP. Typically, they are used directly from the web by:

including the js library

calling out the API (via HTTP protocol)

parse the result and display the result. (sometimes, js library may have standard display library as well).

Here, we included the stripe checkout.js library. We are including all the needed information with that call. The result should look like this:

In this scenario, the credit card number doesn’t even touch the local system. That means, PCI compliance does not apply to this site. The credit card information is handled by Stripe.

Architecturally, applications are converging to this broad pattern:

In this picture, the backend services are exposed by the API’s. With the rise of HTML5 and the front-end MVC, the architecture will look like this:

What it means is this: The API’s can directly be consumed by the browser based application. We do not really need server side page creation at all. For instance, I can develop a static shopping mall application with ability to track users, send mails, take payments, integrate with warehouse, all from within the browser, without writing any server side code!

This paradigm is becoming so successful, there are several companies that are catering to developing, documenting, managing, and delivering the API’s:

apigee: API management and strategy company. They raised close to $107 million dollars so far.Their strategy especially focuses on mobile application development on API’s.

Mashery: Competition to apigee. They only (!) raised $35 million dollars. They have been at this game far longer.

Layer7: They are extending their SOA governance to API management and governance.

Apiary: This company offers services to collaboratively design and develop services. They generate documentation, test services from the API description. They have a nice site, http://apiblueprint.org/ that describes API development and offers several services free.

Apiphany: Acquired by Microsoft, this company is going to serve API management within Azure family.

There are several other companies that have entered this already crowded market. If history is any indication, eventually, the technologies, tools, and skills that these companies are developing will become available for enterprises at competitive prices.

Other industries: How they embracing API’s

These API management companies provide only limited perspective on API development. To truly embrace API based technologies, solution design, we should look at the current generation technology companies. The website http://leanstack.io/ describes how cutting edge technology solutions are built, using API’s offered by several other companies. For instance, highly successful Pinterest uses the following services:

As you can see, several of these cloud services are available as API’s to integrate into applications. Google analytics lets apps track users. Qubole is used for big data services. Sendgrid lets apps send mails.

In the current crop of companies, there are several services that are cheap enough and modern enough for banks to be able to integrate into their applications. They can reduce the effort in developing comprehensive solutions and increase customer satisfaction. For example, Rightsignature offers easy way to get docs signed, with support for integration via API’s. Hubspot provides API’s to make use of its inbound marketing services. Qualaroo lets you design, target, and host surveys for your users easily. Spnnakr lets you offer segmented pricing.

Summary

Banking is evolving. By focusing on the essential services, it can foster new innovations from the community of users and companies. Currently, technology is embracing API’s as a way to integrate services from different providers to create new consumer applications. Banks may not be able to create such an ecosystem by themselves, but they can participate already existing ecosystems. By creating the right technology support via API’s, banks can offer the solutions that meets the needs of diverse audience with different demands on privacy, convenience, security, and trust.

This post has nothing to do with whether “obamacare” is good or bad. It is only about the discussion of the technology stack and the details of it.

At $634M, it is one of the costlier government projects. At its launch, it ended up failing for several users. Even now, they estimate 5M lines of code change to fix the system. What is going on?

The problem, viewed from one angle is simple. Let the users discover and onboard to a particular plan. And, it should cater to large number of users. Since the users do not have other options, you can dictate terms to users (you could say that they need to download a specific version of browser to work with it). Looks easy enough.

On the other hand, the problem is complex. There are any number of plans. There are several different exchanges. The eligibility criteria is complex. There are different agencies, databases, vendors involved. And, the integration is bound to be complex. So over all, it is, well, complex.

To top it off, it is overseen by government agency. These people are good at sticking to rules, procurement, checklists etc. If they check for the wrong things, the site meets all the requirements, and yet, fail.

Tech stack details:

The tech stack is modern. They rely on JS on the browser.

Performance issues

Summary: They are doing lot of rookie mistakes in optimizing the page for performance. With minimal effort, they can take care of most of the issues.

They seem to use 62 JS files in each page. They need to use fewer files and minified as well to reduce the round trip. With 62 files, and that too, without expires headers, we are looking at 15 round trips and that means around 5 seconds of loading time itself (assuming .3 sec for round trip and processing).

The page is heavy! The JavaScript is 2MB and the css is .5 MB and images are .25 MB. So, over all, the site needs to download 2.75MB just to start working.

For the returning user, the situation is only marginally better. They still need to make 85 round trips (that is the number of components); but they only need to download .5 MB.

If experienced folks developed this site, they can reduce the round trip time to less than 1 second (5 fold improvement), easily.

Code quality issues

First the tech stack details.

Bootstrap

JQuery

Backbone

Underscore

JSON

JQuery UI (Why?)

Parts of Scirptaculous

Their stack is not to blame. They are making use of API’s heavily. They use bootstrap (version 2.3), Jquery, backbone, underscore and JSON. I think backbone is too complex a technology (I am partial to Angular, or lot of other modern JS technologies), and the rest are simple enough. In the hands of novice developers Backbone can get very convoluted. In fact, the same can be said off JS.

Let us take look at code quality (what we can see in JS files):

Backbone is complex for these kind of apps. That too for average developers, BB tends to be difficult to use.

Checkout the file: https://www.healthcare.gov/marketplace/global/en_US/registration.js – to understand how the code is laid out. They are not doing template driven development or metadata driven development. This is too much of hard-coded stuff. And, look for “lorem ipsum” too, while you are at it (that shows poor test coverage, or unnecessary code). (this file may be auto generated…).

Use of too many technologies: Shows sub-contracting and no architectural oversight. For instance, if you are using Bootstrap, might as well stick to it, instead of getting JQuery UI stuff. Also, lot of JS files like Carousel etc are built in Bootstrap – why have separate ones any more?

At this point, I don’t have much information: The github pages seem to have some code, but that may not be the one used here. The github uses static html generator (a good strategy) – but that is not what the current website has (Github code seems have been removed now).

Overall, it is looks like high concept, reasonable execution, bad integration, and terrible architectural and design oversight.

We don’t talk numbers enough, in software business. You walk into a store to buy a gallon of milk and you know the price. You ask a software consultant how much the solution costs and he says, “it depends”. You cannot even get him to state assumptions and give a reasonable price.

At IITM, I heard a story about a German professor, and his style of evaluating the exam papers. In one of the questions, the student made a mistake in some order calculation. His answer was accurate but for the the trailing zero. The student got “0”.

Now, in general, if the student understood what he needs to do, and applied the formula, even if made some simple calculation mistake, he would get partial credit, in other courses. Getting zero marks was a shocker to him. But the German professor wanted was for the students to develop a good feel for the answer. For instance, if I were to calculate the weight (er…, mass) of a person and it came to 700 Kg, it should ring some warning bells, right?

In my job, lack of basic understanding about numbers holds back people, unfortunately. This ignorance shows up in places like sizing the hardware, designing the solutions, and creating budgets.

Network Latencies

Suppose you are hosting a web application in San Francisco. Your users are in Delhi. How much minimum latency can you expect?

The distance is around 7500 miles or 12000 Km. Light can travel 186000 miles per second. So, it takes around .04 seconds. But light travels around 30% less speed in fiber than in vacuum. Also, it is not going to travel in a straight line – it may go through lot of segments. Besides, there are relays and other elements that delay the signal. All in all, we can say our signal will have an effective speed of say .15 seconds. Now, we need to a round trip, for any acknowledgement – so that makes it .3 seconds. A simple web page will have 15 components (images, fonts, CSS, and JS). That means around 4 round trips (Most browsers do four components at a time).

So, just in speed of light basis alone, your network is going to take up 1.2 seconds. That is going be the number you are going to add to your testing on your laptop.

Developing Java code

I met a seasoned developer of Java code. He has been developing systems, and well versed with optimization. We were reviewing some code and I noticed the he was optimizing for creating less number of objects.

How much time does it take to create an Object in Java? How much with initialization from String? How much time does it take to concatenate strings? To append to strings? Or, to parse an XML string of 1K size?

The usual answer from most people is “it depends”. Of course, it depends on the JVM, OS, machine, and so many other factors. My response was “pick your choice – your laptop, whatever JVM, and OS that you use, when the system is doing whatever it normally does”. It is surprising that most people have no idea about performance of their favorite language!

In fact, most of us who work with computers should have an idea about the following numbers.

Numbers every should know about computers

These may appear simple enough, but the implications of developing code are enormous. Consider some sample cases:

What if you want to design a system for authentication for all the users in Google (ignore the security side of the question – only think of it as a lookup problem), how would you do it?

If you want to design a system for a quote server for the stock market, how would you speed it up?

You are developing an e-commerce for a medium retailer with 10000 SKU’s (Stock keeping Units). What options would you consider for speeding up the application?

Naturally, these kind of questions lead to some other numbers. What is the cost of the solution?

Cost of computing, storage, and network

There are two ways you can go about constructing infrastructure: the lexis-nexis way or the Google way. Remember that Lexis-Nexis is a search company that lets people search for legal, scientific, and specialty data. Their approach to build robust, fail-safe, humongous machines that serve the need. On the opposite spectrum is Google, which uses white boxed machines, with stripped down parts. I suppose it gives a new meaning to the phrase “Lean, mean machine”. (Incidentally, HDFS etc, take similar approach).

Our situation lies more towards Google. Let us look at the price of some machines.

A two processor, 32 threaded blade server, with 256 GB is around $22000. You can run 32 average virtual machines on this beast. Even if you are going for high-performance machines, you can run at least 8 machines.

If you are going for slightly lower end machines (because you are taking care of robustness in the architecture), you have other choices. For instance, you can do away with ECC memory etc. You can give up power management, KVM over IP etc, for simpler needs. [Example: setting up internal POC’s and labs.]. If that is the case, you can get 64 GB machine, with SSD of 512 GB and 5 TB storage at around $3000.

So, you have some rough numbers to play with, if you are constructing your own setup. What about the cost of cloud? There, pricing can be usage based, and can get complex. Let us take a 64GB machine with 8 hyper-threads. If we are running it most of the time, what is its cost?

Amazon’s cost tends to be slightly on the high side. You pay by the hour and it costs around $1.3 per hr. That is roughly equivalent to $1000 per month. If you know your usage patterns well, you may optimize it down to say, $500 per month.

Or, you could use one of my favorite hosting sites: OVH. There, a server like the above, would cost around $150 per month. Most of the others fall somewhere in between.

Now, do a small experiment in understanding the costs of the solutions: Say, to create an e-commerce site that keeps the entire catalogue cached in memory, what is the cost of the solution?

To truly understand the cost of solution, you also need to factor in people cost as well. That means, what is the effort to develop solutions, operate and support them.

TCO: Total cost of ownership

To understand the cost of operations and support, here is a rule of thumb. If you are automating tasks, reducing human intervention, you can assume that cost of design and development can range from $50 to $250. The variation is due to location, complexity of systems, effort estimation variations, choices of technology, and a few other details.

A few details worth noting:

You can get a good idea of salary positions and skills by looking at the sites like dice.salary.com. Try indeed.com, for a more detailed look at, based on the history of postings.

To get the loaded cost to the solution for labor, multiply the cost per hour by 2. For instance, if you need to pay $100 K salary, the hourly cost is going to be $100 (2000 hrs per year).

I used to run 4 vms on my machine for my application testing. Once I discovered containers, I now run close to 40 or 50 on the very same machine. And, as opposed to 5 minutes that take to start a VM, I can start a container under a second. Would you like to know how?

Virtualization: Using virtual machines

I was using computers since 1982 or so. I came to Unix in around 84 and always was using them until 1997. I had to use Windows after that, because Linux did not have the tools needed for regular office work. Still, because I needed Linux, I turned to an early stage company called VMware.

Now, virtualization is a multi-billion dollar industry. Just about most datacenters virtualized their infrastructure: CPU’s, storage, and network. To support the machine level virtualization, technology developers are redefining the application stacks as well.

Let us see how Virtualization looks normally.

This kind of virtualization offers lot of advantages:

You get your own machine, where you can install your own OS. While this level of indirection is costly, over the time several advances helped:

X86/X64 machine level virtualization: That means, you can run guest OS at native speeds.

Addition of software tools such as VMWare tools: These tools proxy the guest OS requests (I/O, network) directly to the host OS.

You get root privileges, so you can install whatever you want. And, offer the services just like a physical machine would.

Backup, restore, migrate, and other management facilities are easy to manage with standard tools. There are tools to do the same in physical machines as well, but they are expensive, and not so easily doable in self-service model.

But, let us look at the disadvantages also:

It is expensive: If you just want to run a single application, it looks ridiculous to run an entire OS. We made tremendous progress in developing multi-user OS’es –- why are we taking a step back towards single user OSes?

It is difficult to manage: It may be easier compared to managing a physical machine. But, if we are comparing to running an app, it is lot more complex. Imagine: you not only need to run the application, but the OS also.

To put it differently, let us take each of the advantages and see how they are meaningless in lot of situations:

What if we don’t need to run different OS’es? The original reason for running different OS’es was to test apps on different OS’es (Windows came in many different flavors, one for each language, and with different patch levels).

Now, client apps run on the web, a uniform platform. So, no need to test web apps on multiple OS’es.

Server apps can and do depend not on OS, but a different packages. For instance, an application may run any version of LInux, as long as there is Python 2.7, with some specific packages.

Virtualization: Multi-user operating systems

We have a perfect system to run such applications: Unix. It is a proven multi-user OS, where each user can run their own apps!

What is wrong with this picture? What if we want to run services? For example, if we want to run my own mail service? Web service? Without being root, we cannot do that.

While this is a big problem, you can easily solve it. Turns out that most of the services that you may want to run on your machine, mail, web, ftp, etc, can easily be setup as a virtual service, on the same machine. For instance, Apache can be setup to serve many named virtual hosts. If we can setup to provide each other control over their own virtual services, we are all set.

Virtualization: At application level

There are several companies that did exactly that. In the early days of the web, this was how they provided users their own services on the machine. In effect, users were sharing their servers – mail, web, ftp, etc. Even today, most of the static web hosting, or prepackaged web apps run that way. There is even a popular web application webmin (and virtualmin) that can let you manage the virtual services.

What is wrong with this picture? For the most part, for fixed needs, for fixed set of services, it works fine. Where it breaks down is the following:

No resource limit enforcement: Since we are doing virtualization for each application, we have to depend on the good graces of the application to do the resource limit enforcement. If your neighbor subscribed to lot of mailing lists, your mail response will slow down. If you hog the CPU of the webserver, your neighbors will suffer.

Difficulty of billing: Since metering is difficult, we can only do flat billing. The pricing does not depend on the resource consumption.

Unsupported apps: If the application you are interested does not support this kind of virtualization, you cannot get the service. Of course, you have a choice of running the application in your own user space, with all the restrictions that come with it (example: No access to some range of ports).

Lack of security: I can see what all apps all other users are running! Even if Unix itself is secure, not all apps may be secure. I may be able to peek into temp files, or even into memory of the other apps.

So, is there other option? Can we provide a level of control to the users where they can run their own services?

We can do that if the OS itself can be virtualized. That is, it should provide complete control to the users, without the costs of a VM. Can it be done?

Virtualization: At OS level (Containers)

In early days, they had VPS (Virtual private servers), which did provided a limited bit of control. Over the years, this kind of support from OS has become more sophisticated. In fact, there are several options now that elevate the virtual private servers into containers, a very light weight alternative to VM’s.

In the beginning there was chroot to create a “jail” for applications so that they do not see outside of that folder and subfolders is a popular technique to create a sandbox for applications. Features like cgroups have been added to Linux kernel to limit, account, and isolate resource usage to process groups. That is, we can designate a sandbox and its sub processes as a process group and manage them in that way. Lot more improvements which we will describe make the full scale virtualization possible.

Now, there are several popular choices for running these kind of sand boxes or containers: Solaris based SmartOS (Zones), FreeBSD Jails, Linux’s LXC, Vserver, and commercial offerings like Virtuozzo and so on. Remember that within that OS, the kernel cannot be changed by the container. It can, however, overwrite any libraries (see later about union file system).

In the Linux based open source world, there are two that are gaining in popularity: Openshift and Docker. It is the latter that I am fascinated with. Ever since dotcloud opensourced it, there were lot of enthusiasm about that project. We are seeing lot of tools, usability enhancements, and special purpose containers.

I am a happy user of docker. Most of what you want to know about docker, can be found at docker.io. I encourage you to play with docker – all you need a Linux machine (even a VM will do).

Technical details: How containers work

Warning: This is somewhat technical and as such, unless you are familiar with the way OS works, you may not find it interesting. Here are the core features of the technology (most of this information is taken from: Paas under the hood, by dotcloud).

Namespaces

Namespaces isolate resources of processes from each other (pid, net, ipc, mnt, and uts).

Pid isolation means a processes residing in a namespace do not even see other processes.

net namespace means that each container can bind to whatever port it wishes to. That means, port 80 is available to all containers! Of course, to make it accessible from outside, we need to do a little mapping – more later. Naturally, each container can have its own routing table, and iptables configuration.

ipc isolation means that processes in a namespace do not even see other processes for IPC. It increases security and privacy.

mnt isolation is something like chroot. In a namespace, we can have a completely independent mount points. The processes only see such file system.

uts namespace lets each namespace have its own hostname.

Control groups (cgroups)

Control groups, originally contributed by google, lets us manage resource allocation for groups of processes. We can do accounting and resource limiting at group level. We can set the amount of RAM, swap space, cache, CPU etc. for each group. We also can bind a core to a group – a feature useful in multicore systems! Naturally, we can limit number of i/o ops or bytes read or written.

If we map a container to a namespace and the namespace to a control group, we are all set in terms of isolation and resource management.

AUFS (Another Union File System)

Imagine the scenario. You are running in a container. You want to use the base OS facilities: kernel, libraries — except for one package, which you want to upgrade. How do you deal with it?

In a layered file system, you can only create what we want to. These files supersede the files in the base file system. And, naturally, other containers only see base file system and they too can selectively overwrite in their own file system space. All this looks and feels natural – everybody is under the illusion of owning the file system completely.

Security

Lot of security patches rolled into one called grsecurity offers additional security:

buffer overflow attacks

Separation of executable code and writable parts of the code

Randomizing the address space

Auditing suspicious activity

While none of them are revolutionary, taken together all these steps offer the required security between the containers.

Distributed routing

Let us suppose each person runs their own apache, on port 80. How do they expose that service on that port to outsiders? Remember that in the VM’s, you either get your own IP or you get to hide behind a NAT. If you have your own IP, you get to control your own ports etc.

In the world of containers, this kind of port level magic is bit a more complex. In the end, though, you can setup a bridge between the OS and the container so that the container can share the same internet interface, using a different IP (perhaps granted from DHCP source or, manually set).

A more complex case is, if you are running 100’s of containers, is there a way to offer better throughput for the service requests? Specifically, if you are running web applications? That question can be handled by using standard HTTP routers (like ngnix etc).

A famous saying goes that you hire people who are “smart” and “get things done”. I cannot tell you if people can get things done from their innate capabilities or intelligence. But, over time, I realized that most intelligent people that I come across have the similar qualities. I struggled to characterize them until I came across this list in the book “Gödel, Escher, Bach: An eternal golden braid”. Quoting verbatim, here they are:

to respond to situations very ﬂexibly;

to take advantage of fortuitous circumstances;

to make sense out of ambiguous or contradictory messages;

to recognize the relative importance of different elements of a situation;

to ﬁnd similarities between situations despite differences which may separate them;

to draw distinctions between situations despite similarities which may link them;

to synthesize new concepts by taking old concepts and putting them together in new ways;

to come up with ideas which are novel.

I must say that this list is very satisfying. After coming across this list I started using it it my daily life. Am I exhibiting these qualities? Am I evaluating people based on these qualities? Do they correlate with other ways I evaluate the effectiveness of people?

I use this list in multiple ways. I use it to evaluate and improve myself. I use it to evaluate people that potentially work with me. I use it in my activities as an architect, as a technical evangelist, as a coder, as a pre-sales principal, and as a strategist.

To respond to situations very ﬂexibly

I find several people who excelled at school fall short in life later on. People prepare well for well-set idealized scenarios. When the situation differs from what they learnt, they cannot respond flexibly.

For instance, let us see how it works in presales situation. You are prepared for a standard scenario that a customer might face – say, how automation of processes helps to save the money. You go to a customer for whom automation costs lot more than actually the savings that get accrued. If you forget why you were proposing the solution (to save costs), you end up needlessly pushing automation, when all they want is effective reduction of process execution.

In my experience, the flexibility demonstrates the ability to understand the big picture and context of the problem that only comes with a deep understanding of the subject or solution you are trying to present. I find that the following helps me greatly:

Understand the history: History offers a way of understanding subjects and how they come about. If we don’t understand why people are using certain tools or methods, we may not be able to offer a better way. For instance, the why of operating systems offers why we treated files differently from the network connections. Then, we might start thinking why not treat them the same way. We start seeing parallels from history and apply those as the situation demands.

Have a core set of logic tools: While this rule may appear generic, I find that most amount of flexibility in our thinking comes from understanding what is essential and what is not. My set of tools are from model theory – in particular higher order models. I start seeing things from the perspective of completeness and consistency, even when the situation presents in a new guise.

To take advantage of fortuitous circumstances

I am not sure if I agree that this is an essential quality of intelligence. But, still, most successful people have the ability to see opportunities that others don’t. I think it takes courage, self-confidence, desire, and deep sense of conviction all of which are not necessarily attributes of intelligence.

I am also fairly sure that I am not so good in this area. I do not know how one can “improve” here. I suppose this is some innate capability people have, either by nature or nurture.

To make sense out of ambiguous or contradictory messages

I read science fiction stories where species communicate with precision, the exact emotion or facts. I read about conlangs (constructed languages) that are incredibly precise. But, in reality, we deal with lot of ambiguity, imprecision, and sometimes deliberate obfuscation.

Most of us deal with this ambiguity daily. We find an organization pursuing unclear and multiple conflicting paths. We find people making contradictory statements. We may not see consistency in action and word.

It may be fodder for sitcom or science fiction humor – a fish out of water story of foreigners, extra-terrestrials not possessing the verbal intelligence of natives. The consequences for an organization are more disastrous: infighting, multiple directions, and inaction. It forces leaders to constantly explaining people what to do.

While it is easy to dismiss that it is innate capability, I think there is a way to develop it well, in particular, the following skills, which are strongly correlated to this capability:

Reading widely helps: In particular, I think reading good fiction, poetry, and even nonfiction helps us understand the ambiguity and imprecision in language. Reading classics has an advantage as they were widely interpreted from different perspectives.

Writing well helps: Right kind of writing can help cut through ambiguity. When I don’t understand a subject well, I try writing, just as a way of clarifying it for myself.

Understanding of logic helps: In particular, I go back to my first discipline, model theory and higher order logic to understand the model in terms of consistency and completeness.

To recognize the relative importance of different elements of a situation

One of the big milestones in growing up is understanding cause and effect. That feedback loop helps us to know the consequence of our actions. The same knowledge, when applied gives a way to assessing the importance of different elements of a situation.

Fortunately, this is easy enough to deal with. At the risk of generalizing and simplifying, the best way to deal with is through metrics and goals. If you know the end goal, and the impact of the tasks on the ability to reach the end goal, you know how to prioritize. For a full discussion of the topic, see: http://bit.ly/J8b3BW.

To ﬁnd similarities between situations despite differences which may separate them

Abstraction of the core essential elements from different situations is a key human trait. Right kind of abstractions can help people understand the core situations. For instance, if we want to solve a problem, using the right abstraction, we can bring tools and processes from similar solutions.

Even in presales, customers would love to know how you solved similar problems for others. Everybody knows that exact problem is difficult to find. Finding a useful abstraction is key to creating a similar set of problems that can help solve the problem at hand effectively.

To draw distinctions between situations despite similarities which may link them

The flip side of abstraction is reification or concretization. Every problem is different. Only when we understand the differences between the two different problems, we can understand what kind of customization we need to do for this problem. It may be a simple change to a tactic; for being aware of such need is key to look for such change.

To synthesize new concepts by taking old concepts and putting them together in new ways

There is a saying: “There is nothing new under the sun”. But, then, we see new creations from human ingenuity come every year. Most of them are novel applications of existing ideas. In my own field of programming languages, I see the ideas from 80’s becoming popular now. When I see CPS (Continuation Passing Style) in Javascript, I am home work assignments from my student days.

Creation of new concepts from old takes several pieces of puzzle:

Good understanding of history: Why this problem is important, what techniques were tried, what was the genesis for the final solution – all of these are important when we need to put together a new solution based on the old concepts.

Good understanding of the changes in situations: For instance, when we were solving main memory databases in 90’s, we had maximum of 4GB in general machines. These days, I have 64GB on my desktop machine. Naturally, the assumptions have changed, opening up the solutions that were not considered before. For example, changing economics of printing changed the way books are published.

Good understanding of the old concepts: Unfortunately, learning, especially in computers has become more and more understanding the details instead of concepts. Details will tie us down to an existing way of thinking. Understanding the core concepts (why, how, and what) lets us put them together in new ways.

To come up with ideas which are novel.

I suppose, this is what classically considered intelligence and genius. I do not have any suggestions on how one goes about improving in this area. I find that I am a pastiche kind of person – I take a concept and apply in a different area. Personally, I do not think I have any ideas that can be called truly novel.

Concluding remarks

Looking at the core traits of intelligence, I can only conclude that some of them are hard to acquire. I do not know how one can train oneself to come up with new ideas. But, as for other traits, there is hope that we can practice deliberately to hone those skills. Fortunately, the world is large; there are lot of ideas coming from different places. All you need is to pick a set of tools that are good enough; and master them; and apply them consistently to learn these skills.