"The system stores over 800 million objects (an object = a user event such as receiving an email or logging into IMAP) within Solr and 9.6 billion within Hadoop, which equals 6.3 TB compressed.

[...] For example, we wanted to build a tool that would allow our customers to search their logs directly. We had been keeping an eye on the Apache Hadoop project since its inception, and were extremely impressed with its progress and direction. Hadoop is an open-source implementation of Google File System and MapReduce... a system that is designed specifically for large scale distributed data processing. It scales out it's workload horizontally by adding servers and distributing the data and MapReduce jobs amongst the servers. Other companies were already using it for their own log processing. So chose to go with Hadoop. In about 3 months we build a fresh new log processing system using Hadoop, Lucene and Solr."

"Think back to that band of chimps watching one of its members learn out how to use a termite stick. With humans, as with chimps, it's monkey see, monkey do. We learn by watching and imitating. For most of human history, humans learned how to use tools by watching other humans use those tools, then copying the behavior.

But it's a funny thing. Although we feel hyperconnected in the era of networked communication, it turns out that we have surprisingly few chances to watch and imitate how other people use their software tools and information systems.

When we're sitting side by side, we can look over each others' shoulders, and watch, and learn. But for decentralized teams, voice and text are still the dominant modes of communication, and they don't enable that same kind of direct transfer of knowledge and experience.

Screensharing can mediate that direct transfer synchronously, in realtime. For that reason, I think it's critical for us to get to the point where it's just as trivial to share screens and keyboards remotely as it is to set up a text or voice session.

One reason why this matters has to do with tacit, or unconscious, knowledge. Things we know how to do, but don't really know that we know, and can't articulate or explain."

"When the Java Content Repository (JCR) standard first came out it was supposed to bring in a new era of compatibility between content repositories and put an end to the content silo. There was, and still is, a lot of talk about it and just about everyone added JCR compliance to their marketing materials. [...] There are a few CMSs around that do have good JCR support - Alfresco for example - but they're few and far between and even with that, there isn't a lot of people taking advantage of that support and the standardization of the repository interface.

Then along came Atom which is all about remote access and manipulation of data and missing probably 90% of the functionality that JCR offers. It really isn't a competitor to JCR at all and yet it's doing more to break down content silos than JCR ever has.

[...] Having Atom support in your product, serving and consuming as necessary is becoming an extremely powerful feature."

"I love the way MarkMail gives me a bunch of drill-down choices in the UI, and as I choose them, rewrites the command-line in the search box. I'd love to see features like this in my other mail packages. With Mail.app on Mac OS X, for example, it's impossible to do a complex search. You can search for a text string in the from field, the subject line, or the entire message, but what if you want to say "I want a message on x, from Joe, to Mary, sent between April and June of 2006." Even on Gmail, where I can do this kind of search with Search Options, I have to go to another whole screen, out of the search flow, to do it. You can construct that kind of a search in MarkMail just by navigating around. Yumm. How long before regular mail vendors start doing this kind of thing? This is a really sweet search interface."

"A big project I’m peripherally involved with needed to include an outward-facing Wiki, and I suggested that MediaWiki was damn good stuff. They put in quite a bit of work and failed to get MW to integrate with the rest of the system. Yes, it’s a good wiki, but it shouldn’t have been that hard to make it play nice with others, and I got the impression that the PHP-ness of it was a big part of the problem."

I'm not sure where PHP gets in our way when we're trying to write good software...

"When we were still doing client work back in the early 2000s we got a lot of calls about designing intranets. Everyone wanted an intranet. The employees we talked to loved their intranets too—their jobs depended on having access to bits of information, files, forms, etc. that were only available on their company intranet.

But since then I haven’t heard much noise from the intranet camp. Have people given up because the options were too complicated? Or have they built their own? Or are they piecing together blogs and wikis and file storage services and online calendars to get the job done?"

"RFC 2068 is about a week shy of being 11 years old. Is the 255 byte URI length recommendation still applicable? From the client perspective, possibly not. But what about from the proxy perspective? And are there clients (possibly mobile browsers?) that I’m not taking into consideration that still impose a limitation on the URI length?

"Why not? Because, as far as I can make out, the idea that we will all be better off if we pretend that XML Schemas is a unified and whole specification, one size that can fit all, then somehow it will magically happen. But fantasy is a really poor substitute for reality. Time and time again I have seen clients happy about XML Schemas and its promises, only to have their hopes dashes as they realize that as soon as they need to start deploying they have to use subsets and there is no support from “standards” to help interoperability."

"Usually the things that make or break a project are process and people issues. The way that you work on a day-to-day basis. Who your architects are, who your managers are, and who you are working with on the programming team. How you communicate, and most importantly how you solve process and people problems when they come up. The fastest way to get stuck is to think that it's all about the technology and to believe that you can ram your way through the other things. Those other things are the most likely ones to stop you cold.

In my first jobs, I saw lots of managers making stupid decisions, and so, logically, I came to the conclusion that managers and management was stupid. It's a commonly held belief in our profession: if you're not smart enough to deal with the technology, you go into management. Over time I very slowly learned that the task of management wasn't stupid, it's just very, very hard. That's why all those stupid decisions are still being made; management is much harder than technology because it involves virtually no deterministic factors. It's all guesswork, so if you don't have good intuition you'll probably make stupid decisions.

[...] There's one more very important maxim from Gerald Weinberg which doesn't really answer anything as much as gives you a way to understand what happens. He says: "Things are the way they are because they got that way ... one logical step at a time." It's the legendary frog in the saucepan. So from your fresh new perspective things might look ridiculous, but remember that each decision on the way was made by someone weighing the issues and making what seemed like the best choice at the time. This viewpoint doesn't solve the problem but it can make you more compassionate about the people who are stuck there."

"Typically, when new prospects first visit your site, you're simply one of ten sites on the SERP [Search Engine Results Page]. The only way they'll shortlist your site is if you can convince them in two minutes.

To determine how much complexity you can afford in a user interface, you must analyze user engagement levels: Do they care deeply, or do they just want to get something done as quickly as possible? Typically, users care less than you think! You're not important to them. This is one of the main reasons companies need systematic usability studies: to make explicit the fact that outside customers don't find your design as important as you do (because you work on it all year)."

"It helps tremendously to work out the ‘chatter’ between the client and the server in human language before designing it. Here is an example of RESTful chatter between a client inventory application, and it’s corresponding server application: [...]

The point of the chatter is to ensure that the client is passing a complete transaction with every request, and that the server passes a complete transaction with every response. Similar to the CRC cards for OO design, the Chatter ensures that you have your states, transactions and interfaces well defined."