Cloud Development and Alien Abductions

Navigation for Clemens Vasters - Technology|FABRIQ

We've built FABRIQ, we've built Proseware. We have written seminar series about Web Services Best Practices and Service Orientation for Microsoft Europe. I speak about services and aspects of services at conferences around the world. And at all events where I talk about Services, I keep hearing the same question: "Enough of the theory, how do I do it?"

Therefore we have announced a seminar/workshop around designing and building service oriented systems that puts together all the things we've found out in the past years about how services can be built today and on today's Microsoft technology stack and how your systems can be designed for with migration to the next generation Microsoft technlogy stack in mind. Together with our newtelligence Associates, we are offering this workshop for in-house delivery at client sites world-wide and are planning to announce dates and locations for central, "open for all" events soon.

If you are interested in inviting us for an event at your site, contact Bart DePetrillo, or write to sales@newtelligence.com. If you are interested in participating at a central seminar, Bart would like to hear about it (no obligations) so that we can select reasonable location(s) and date(s) that fit your needs.

Finally, finally, finally. It was a looong wait. As many others, we were in a wait loop for WSE 2.0 for a long time and that let us do what we do today only much, much later than we initially anticipated. So after being able to test on and adjust for the WSE 2.0 RTM bits for the last four weeks, we're now happy enough with our "1.0" that we're ready to share it:

This is a proof-of-concept project collaboratively created by Microsoft EMEA and newtelligence AG. We have tested intensively for quite a few sets of use-cases, but this is not a product. We are giving this to you because we think it's very useful architecturally and most implementation aspects isn't too bad either, and we do expect you to play with it. We don't give it to you to install this in a production environment tomorrow or even on the day after.

The support policy for FABRIQ is very simple: There is none. If you download this, you are absolutely and entirely on your own, legally speaking. We are keen to hear your feedback and are curious whether and for what you find this useful, but this is no product and therefore there's no support whatsoever. (If you find this so useful that you want customization, support, or need help to get this from near-production quality to production-quality, sales@newtelligence.com is a great place to write e-mail to)

This is "work in progress" and you are getting a version that is not considered finished. You will find artifacts in the code that are not (anymore or yet) used. You will find code branches that we not (anymore or yet) hit. There are a few places, where we cut some corners in terms of implementation efficiency in order to get this out early. You will find that there is a bit of a disconnect between the specification documents that we have in the package vs. the documentation that you'll find and we could have done a better job cleaning it all up. We love this ourselves and will continue to polish it.

Contributions: We give you the code and you can use it and change it. For the first version and the next minor drops, we'll not have a public code repository that people can check things into immediately, because the beast turned out to be so complex that we need to stay in control for a little while. If we allowed "random" community contributions early, people who don't live inside in the codebase could too easily seemingly unrelated stuff. Therefore: If you want to change or add stuff, wrap up your changes along with a good reason why that's needed and send it here.

Discussions: Write what you like or hate or what you don't understand into the forums in the workspace or just blog about it and refer to this entry or relevant entries on my blog or Arvindra's blog once he's fully set up. We'll accept everybody into the workspace; just apply and you'll be granted access as soon as someone sees it.

Credit where credit is due: Very many thanks to the development team in Argentina, with Eugenio Pace, Adrian Nigro, Federico Winkel, and Juan Carlos Elichirigoity, who have worked very very hard turning my "written in two weeks in a hurry" prototype code into something that's actually useful.

Before I can get into explaining how the FABRIQ works and how to configure it, I need to explain a bit of the terminology we use:

A network is the FABRIQ term that's rougly equivalent to an "application". A network consists of an arbitrary number of network-distributed nodes that are running inside the scope of the network. The network creates a common namespace for all of these nodes. Networks are configured using a single XML configuration document that is submitted (or routed via another network) to all hosts that shall host the network's nodes.

A node is the FABRIQ term that is roughly equivalent to a "service" or "component". A node is the smallest addressable unit. Every node has a "relative node URI" that is composed of the network name and the node's own name into {network/node}. This relative node URI can be composed with absolute, transport dependent URIs such as http://server/vdir/network/node or msmq://machine/queuename/network/node. Within a network, the runtime is also capable of resolving logical addresses of the form fabriq://network/node and automatically map them to physical addresses. At runtime, a node accepts messages and dispatches them into one of one or more action pipelines. Each node may be guarded by a set of WS-Policy assertions, including Kerberos and X.509 cert authentication and authorization claims. A node may be hosted on a dedicated machine, one a well defined set of machines or on "any" machine within a cluster.

An action pipeline is a pipeline that is associated with an action identifier and is roughly equivalent to a "method". An action identifier is a URI as per WS-Addressing's definition of wsa:Action and is mapped to SOAPAction: whenever we go through HTTP. A node must host at least one action pipeline with no limit on the number of action pipelines it can support. An action may declare a set of message schema-types that it understands and those message definitions may be used for validation inbound messages. An action has one or more outbound message routes that are matched against the result message action or destination. Multiple routes may match a message, which causes the message flow to fork. For each route exist one or multiple prioritized routing destinations. If multiple destinations have the same priority, the engine will balance calls across those, otherwise the engine will use the ones with lower priority as backup routes. At the end of each action pipeline is a sender port that sends resulting messages out to their destinations, which may be other FABRIQ nodes or any other external endpoint that understands the respective one-way message being sent.

A pipeline is a composition of a sequence of handlers or nested pipelines. Pipelines can be nested in arbitrary depth. Pipelines are strictly unidirectional message processors that have no concept of a "response" on the same thread analogous to a return value (hence all actions are one-way only). A pipeline may or may not be based on a predefinable pipeline-type. Pipeline-types allow the definition of reusable pipelines that can be reused within the same network or (via import) in multiple networks.

A handler refers to a software component (a CLR class) implementing a set of interfaces that allow it to be composed into and hosted in a pipeline. Handlers should be designed to perform only very primitive operations that can then be composed into pipelines to implement specific functionality. Built-in handlers include a content-based routing handler and an XSLT transformation handler. Custom handlers may contain any type of logic. A handler receives messages and may consume them, evaluate and annotate them and yield any number of resulting messages. The definition of a handler embeds an XML fragment that allows the handler to configure itself. The actual reference to the CLR class implementing the handler is defined in a handler-type.

A handler-type associates a CLR class with a name that can be used to define handlers within a configuration file. It also allows the declaration of a code-base URL for the CLR class. This feature allows the installation of "virgin" FABRIQ runtimes in a cluster and have the runtimes auto-download all the required code for hosting a node from a central code store and therefore dramatically eases deployment and dynamic reconfiguration of a FABRIQ cluster.

In the next couple of postings I will map these terms to concrete config files.

The interesting bit about config is that FABRIQ's configuration mechanism uses the FABRIQ itself. FABRIQ has a predefined (extensible, configurable) network "fabriq" with a node "configuration" that currently defines a single action "configure". The pipeline for that action consists of a single handler (the FabriqConfigurationHandler) and that expects and accepts the configuration files I'll describe over the next days as the body of a message. With that, the configuration mechanism can be secured with policy, or can be embedded into a larger network that does preprocessing or even performs automatic assembly of configuration, or that automatically distributes configuration from a single point across a large cluster of machines.

Achim and myself are currently in a series of very quick rev-cycles for the first public release of the Microsoft/newtelligence FABRIQ project that we did with and for Microsoft EMEA HQ and that was conceived, driven and brillantly managed by my architect colleague Arvindra Sehmi, who gave me the lead architect role for this project.

[Reminder/Disclaimer: this is not a product, but rather a pretty elaborate "how-to" architecture example that comes with an implementation. Hence it's not a supported Microsoft or newtelligence "framework" or an attempt at some general, definitive guidance on how to write services. FABRIQ is an optimized architecture for fast, one-way, message processing within network-distributed nodes consisting of sequences of dynamically composed primitive processing steps. This isn't even trying to get anywhere near the guidance aspirations of Shadowfax, or let alone all theguidance we're getting from the Indigo team or even the parallel work I've been doing for MS by building Proseware.]

We've settled on build 1.0.4173 (yesterday) to be the TechEd version, but we still found a last minute issue where we weren't using WSE 2.0 correctly (not setting the SoapEnvelope.Context.Destination property for use with a bare WSE2 Pipeline in the presence of policy) and when I reassembled the distribution I didn't reset an option that I use for debugging on my machine and that caused installation hiccups over at Achim's machine. Achim commented the hour-long bug hunt with "Ah, you gotta love software!".

There will be hands-on labs at TechEd Europe led by Achim and Jörg that let you play with what we (very much including our friends at Microsoft Argentina and Microsoft EMEA) have built. And even if you don't have a proper use for a one-way queuing network architecture, it actually turned into a fun thing to play with.

I'll be starting to explain aspects of the spec over the upcoming days and will explain how the architecture works, how you configure it and what its potential uses are. Already posted is some relevant information about the great idea of an XmlReader-based message design (which I designed inspired by the Indigo PDC build) and our use of lightweight transactions.

I am in the boot phase for the next software project right now (proprietary work) and I have identified very many good uses for the FABRIQ model in there already (hint).

Once all parties involved are giving their "thumbs up", we'll also make the source code drop and the binaries available to the public (you) and from there we're looking forward to your input (and contributions?).

Proseware, Inc. is an online bookseller. A big one. They have warehouses all over Europe, have millions of customers, sell millions of different items and process tens of thousands of orders every day. So imagine they came around to you and asked you to build a system for them. What would you do? In this session, we explain what we would do and what we did. The Proseware Services Demo consists of 14 autonomous, collaborating services, designed with best-practice architecture principles and implemented on Windows Server 2003. Proseware leverages the power of technologies such as Enterprise Services, ASP.NET Web Services, the Microsoft Message Queue, Microsoft Web Services Enhancements 2.0 to implement a large scale, robust, secure and scalable service oriented system that shows how all these technologies can be put to work effectively and that reflects the complexity of real-life enterprise applications.

And this is going to be great fun, too:

ARC230 The Nerd, the Suit and the Fortune Teller (sharing the stage with Pat Helland and Rafal Lukawiecki)

Object Orientation promised to deliver us from all IT evil and to ensure longevity and reuse of software. With today’s business requirements changing faster than it takes to compile an application and an ever-present call for integration, even that approach does not seem to work. However, it seems that Service Oriented Architecture (SOA), Operational Service Orientation of IT departments (MOF, ITIL), Service-Based Interoperability (WS-Guidelines) and Service-Based User Interfaces (messenger, wizards) are quietly converging onto a new paradigm in IT. For the lack of a better name, we call it “Service Oriented Convergence”. Come to this session and see if this concept has any merit and value to you. Rather than endure a typical PowerPoint presentation, you will observe a discussion between: an experienced developer, an unforgiving businessman in charge of IT and a visionary technology innovator. On your behalf, they will battle out their differing points of view and leave you with useful guidance on ways to handle this very important issue that will affect your job, work and future. Oh, and we hope this session is unlikely to be boring.

There is also an updated sessions on the FABRIQ (ARC405, with Arvindra Sehmi) and even a hands-on lab where you can play with the FABRIQ yourself (ARC-IL01, led by newtelligence instructors Achim Oellers and Jörg Freiberger).

I am back home from San Diego now. About 3 more hours of jet-lag to work on. This will be a very busy two weeks until I make a little excursion to the Pakistan Developer Conference in Karachi and then have another week to do the final preparations for TechEd Europe.

One of the three realy cool talks I'll do at TechEd Europe is called "Building Proseware" and explains the the scenario, architecture, and core implementation techniques of Proseware, an industrial-strength, robust, service-oriented example application that newtelligence has designed and implemented for Microsoft over the past 2 months.

The second talk is one that I have been looking forward to for a long time: Rafal Lukawiecki and myself are going to co-present a session. And if that weren't enough: The moderator of our little on-stage banter about services is nobody else than Pat Helland.

And lastly, I'll likely sign-off on the first public version of the FABRIQ later this week (we had been waiting for WSE 2.0 to come out), which means that Arvindra Sehmi and myself can not only repeat our FABRIQ talk in Amsterdam but have shipping bits to show this time. There will even be a hands-on lab on FABRIQ led by newtelligence instructors Achim Oellers and Jörg Freiberger. The plan is to launch the bits before the show, so watch this space for "when and where".

Overall, and as much as I like meeting all my friends in the U.S. and appreciate the efforts of the TechEd team over there, I think that for the last 4 years TechEd Europe consistently has been and will be again the better of the two TechEd events from a developer perspective. In Europe, we have TechEd and IT Forum, whereby TechEd is more developer focused and IT Forum is for the operations side of the house. Hence, TechEd Europe can go and does go a lot deeper into developer topics than TechEd US.

There's a lot of work ahead so don't be surprised if the blog falls silent again until I unleash the information avalanche on Proseware and FABRIQ.

I am getting ready for the Longhorn Developer Preview tour. Now that the whole notebook ordeal is hopefully over, I have been and still am polishing slides and we'll have an online rehearsal today during the day. Furthermore, we're working with Microsoft EMEA on a two day workshop about writing service oriented applications that consolidates all the thinking that I've been blogging about in the past year. The "sample" around which the workshop will center is, not very surprisingly, the FABRIQ.

Slowly, slowly I am seeing some light at the end of the tunnel designing the FABRIQ. It’s a very challenging project and although I am having a lot of fun, it’s really much harder work than I initially thought.

The obvious problem with one-way pipeline processing (and a problem with the composition of independent cross-cutting concerns in general) is that failure management is pretty difficult. Once one of the pipeline components fails, other components may already have done work that might not be valid if the processing fails further down through the pipeline. The simplest example of that is, of course, logging. If you log a message as the first stage of a pipeline and a subsequent stage fails, do you want the log entry to remain where it is? The problem is: it depends. So although you might need to see the message before it is being processed by stages further down the pipeline, you can only find out whether it is flagged as success or failure once processing is complete or you may want to discard the log entry altogether on failure.

Before I go into details, I’ll clarify some of the terminology I am using:

·A message handler is an object that typically implements a ProcessMessage() method and a property Next pointing to the handler that immediately follows it in a chain of handlers.

·A pipeline hosts a chain of message handlers and has a “head” and a “tail” message handler which link the pipeline with that chain of handlers. The pipeline itself is a message handler itself, so that pipelines can be nested inside pipelines. The FabriqPipeline is a concrete implementation of such a pipeline that has, amongst other things, support for the mechanism described here.

·A message is an object representing a SOAP message and has a collection of headers, a body (as an XmlReader) and a transient collection of message properties that are only valid as long as the message is in memory.

·A work set is a lightweight, in-memory 2PC transaction that provides really only the “atomicity” and “consistency” properties out of the well-known “ACID” transaction property set. “Durability” is not a goal here and “isolation” sort of guaranteed, because messages are not shared resources. If external resources are touched, isolation needs to be guaranteed by the enlisted workers. A worker is a lightweight resource manager that can enlist into a work set and provides Prepare/Commit/Abort entry points.

Whenever a new message arrives at a FabriqPipeline, a new work set is created that governs the fault management for processing the respective message. The work set is associated with the message by creating a “@WorkSet” property on the message that references the WorkSet object. The pipeline itself maintains no immediate reference to the work set – it is message-bound.

The FabriqPipeline does not enlist any workers into the work set directly. Instead, message handlers enlist their workers into the work set as the message flows through the pipeline. A “worker” is an implementation of the IWorker interface that can be enlisted into a work set as a participant. Because the pipeline instance along with all message handler instances shall be reusable and shall be capable of processing several messages concurrently, the worker is not implemented on the handler itself. Instead, workers are implemented as a separate helper class (FabriqWorker). Instances of these worker classes are enlisted into the message’s work set. The worker instance gets a reference to the message it deals with and to the handler which enlisted it into the work set; once the worker is called during the 2 phase commit protocol phases, it calls the message handler’s implementation of Prepare/Abort/Commit.

This way, we can have one “all in one place” implementation of all message-handling on the message handler, but are keeping the transaction dependent state in a very lightweight object; therefore we can share the entire (likely complex) pipeline and handlers for many concurrent transactions, because none of the pipeline is made dependent on the message or transaction state.

When a message flows into the pipeline, all a transactional message handler does when it gets called in ProcessMessage() is to enlist its worker and return. If the handler is not transactional, it must never fail (such things exist), can ignore the whole work set story and simply forward the message to the Next handler. So, in fact, a transactional message handler will never forward the message in the (non-transactional) ProcessMessage() method.

One problem that the dependencies between message handlers create is that it may be impossible to forward a message to the next message handler in the chain before the message is processed; at least you can’t make a Prepare==true promise for the transaction outcome until you’ve done most work on the message and have verified that all resultant work will very likely succeed. Messages may even be transformed into new messages or split into multiple messages inside the pipeline, so that you can’t do anything meaningful until you are at least preparing.

The resulting contradiction is that a transaction participant cannot perform all work resulting from on a message before it is asked to commit work, but that message handlers following in the sequence may not have received the resulting message until then and may not even be enlisted into the transaction.

To resolve this problem, the FABRIQ pipeline’s transaction management is governed by some special transaction handling rules that are more liberal than those of traditional transaction coordinators.

·During the first (prepare) phase of the 2-phase commit protocol, workers may still enlist into the transaction. This allows a message handler to forward messages to a not-yet-enlisted message handler during the prepare phase. The worker(s) that is/are enlisted by a subsequent handler because the currently preparing message handler is forwarding one (or multiple) messages to it, is/are appended to the list of workers in the work set and asked to prepare their work once the current message handler is done preparing. We call this method a “rolling enlistment” during prepare.

·Inside the pipeline, messages are considered to be transient data. Therefore, they may be manipulated and passed on during the Prepare phase, independent of the overall transaction outcome. The tail of the transaction controller pipeline (which is the outermost pipeline object) always enlists a worker into the transaction that will only forward messages to outside parties on Commit()and therefore takes care of hiding the transaction work to guarantee isolation.

·Changes to any resources external to the message (so, anything that is not contained in message properties or message headers) must be guarded by the transaction workers. This means that all usual rules about guarding intermediate transaction state and transaction resources apply: The ability to make changes must be verified by tentative actions during Prepare() and changes may only be finally performed in Commit(). In case the external resources do not permit tentative actions, the Abort() method must take the necessary steps to undo actions performed during Prepare().

Whenever new messages get created during processing, the message properties (which hold the reference to the work set and, hence, to the current transaction) may be propagated into the newly created message, which causes the processing of these messages to be enlisted in the transaction, or a new or no work set can be created so that further processing of these messages is separate from the ongoing transaction. That’s what we do for failure messages.

During prepare, participants can log failure information to a message property called “@FaultInfo” that contains a collection of FaultInfo objects. If message processing fails, this information is logged and is, if possible, relayed to the message sender’s WS Addressing wsa:FaultTo, wsa:ReplyTo or wsa:From destination (in that order of preference) in a SOAP fault message.

For integration with “real” transactions, the entire work set may act as a DTC resource manager. If that’s so, the 2PC management is done by DTC and the work set acts as an aggregating proxy for the workers toward DTC. It collects its vote from its own enlistments and forwards the Commit/Abort to its enlistments.