Topics

Featured in Development

Peter Alvaro talks about the reasons one should engage in language design and why many of us would (or should) do something so perverse as to design a language that no one will ever use. He shares some of the extreme and sometimes obnoxious opinions that guided his design process.

Featured in AI, ML & Data Engineering

Today on The InfoQ Podcast, Wes talks with Katharine Jarmul about privacy and fairness in machine learning algorithms. Jarul discusses what’s meant by Ethical Machine Learning and some things to consider when working towards achieving fairness. Jarmul is the co-founder at KIProtect a machine learning security and privacy firm based in Germany and is one of the three keynote speakers at QCon.ai.

Featured in Culture & Methods

Organizations struggle to scale their agility. While every organization is different, common patterns explain the major challenges that most organizations face: organizational design, trying to copy others, “one-size-fits-all” scaling, scaling in siloes, and neglecting engineering practices. This article explains why, what to do about it, and how the three leading scaling frameworks compare.

Eric Evans and Brian Foote discuss the state of Software Design

Bio

Eric Evans (Creator of Domain-Driven Design)leads Domain Language Inc., which coaches and trains teams to make their development more productive and relevant through effective application of domain modeling and design. Brian Foote (Big Ball of Mud, Patterns Languages of Program Design)is a software developer and scholar interested in object-oriented programming, reflection and design patterns.

About the conference

Software is changing the world; QCon aims to empower software development by facilitating the spread of knowledge and innovation in the enterprise software development community; to achieve this, QCon is organized as a practitioner-driven conference designed for people influencing innovation in their teams: team leads, architects, project managers, engineering directors.

Eric: Well I think the reason that we were, I was asked to come was probably because I do talk a lot about strategic design as I call it and how to deal with large systems and bringing sort of high design concept into realistic situations where things are a bit messy and so that’s basically what I was talking about and, Brian?

Brian: I think I wound up here because few years back, we wrote this paper called Big Ball of Mud a few years ago, which said despite our pretensions to the contrary, the entire industry, this came out of the patterns community during the ‘90s which is where Eric and I first met; when you really look at stuff, it was sort of an Emperor’s New Clothes paper, what is really under there is nothing but mud, it’s slushy.

You know, the observation I mean at one point was that if programs are like buildings then most of what we’re building is really more like a shanty town than like the Empire State Building or the Brooklyn Bridge that we’re down the street from here; it’s you have a squalid, there’s a lot of make do, you know, people have to be masters of everything; it’s nothing like what the real architects described; I think over the years, there’s been a lot of recognition that we haven’t really been telling the truth about what’s under there from the standpoint of architecture, which I think is why we both, you know, disavow any connection with real architects.

Eric: Yes; well yes, the disconnection between what architects describe and what’s really there is so great that it makes me, it always made me feel that what they’re doing was irrelevant, much of what they’re doing; obviously the things described as architecture runs a tremendous range and so we can’t speak generally about it.

So whereas I was very impressed by that big ball of mud thing and thought a lot about it and another thing that affected me was that I was on projects where we were doing legacy replacement and the idea was well now we have for example all these advanced technology; remember this was the mid-‘90s and this was the latest thing; and we had these terrible legacy systems that we would just sweep away with our vastly superior and carefully designed new system and that just never worked out for various reasons, which over time I thought I started to have some insight into.

One of the reasons that I’ve always said strategic design instead of architecture is because I think it’s a better description of what you need to do; in order to accomplish goals in an organization, you need a strategy and your design needs to reflect a strategy not just tactical object, biologic view; and so that’s why we’re here I suppose is those respective viewpoints.

Brian: Of course in your talk, I was kind of noticing and I see this in a lot of places these days, this sort of war on mud, I mean I think anyone who looks at software and sees that it’s really this growing organic material winds up deeply conflicted; I mean it just is that way; I mean that winds up being the fundamental truth and in, you know, the 15 years since we wrote about that, one of the things that’s amazed me the most is no one has ever disputed the premise, they just kind of go yes, yes, that’s what we’re dealing with; and there’s really something kind of downbeat about that in a way; even in your talk, I was noticing there is a sense of we will concede most of the terrain and our strategy will be to focus on certain parts of it.

Which is, you know, in some ways, a rather shocking and depressing concession.

Eric: Well I think it is a little shocking and depressing only if we have that preconceived notion of what software ought to be some kind of gleaming crystal tower right; but like in fact I think that you have to concede most of the terrain so to speak to more informal processes and less tightly controlled approaches and most of the time I think it’s fine; I think 90% of what we do with software doesn’t really require any fancy design but some of it does; then if you’re trying to solve some intricate logic problem and you know that you just won’t get it right unless you have crystal clarity, unless you really refine the abstraction so that they fit the problem perfectly, those kind of really classic domain modeling kind of problems and especially when they’re really critical to the business strategy of the moment in some organization then those things just don’t co-exist with that kind of free for all, kind of everything is a little bit compromised kind of view; and those two things just don’t go together.

What I’ve found is that what works well is to create a boundary within the software and I call that context boundary and words mean certain things inside the boundary; they may not mean the same thing outside the boundary and things are orderly with inside the boundary; and when that is created as a kind of small space within the larger, I’ve taken to calling this a bubble, a kind of a bubble; now this is even more modest in ambition than what I described in the book; I mean in the book I’ve talked about establishing context boundaries and being reasonably humble in the face of legacy systems and so forth; but I’ve become even more focused I think on keeping the scale modest and not trying to transform the entire epic system because (a) I don’t think it’s practical goal and (b) I don’t think it’s actually necessary to accomplish our real purpose; I think usually we can create a lot of value without doing that and it’s more a distraction, it’s a denial of the reality of the nature of big systems so.

Brian: So if I wanted New York to better, you wouldn’t raise the entire city, you would gentrify it one neighborhood at a time.

Eric: Well we’d --

Brian: And each would have it, yes, say we were sitting here in Brooklyn, yes.

Eric: Right.

Brian: And each neighborhood will have this local character and its own way of looking out at the world but then how are those neighborhoods going to connect and interact.

Eric: Right; there is some need; also even if you want to cause a neighborhood to gent or if you want to encourage a neighborhood to gentrify, there are things that you do; I mean I was in the Bay Area during the ‘90s and I saw sort of mountain view doing this in a very systematic way; they had a downtown but it had kind of declined and they had a chance to put a new light rail station in and then they used that opportunity to – you know, they put it right in the middle of that downtown area; they zoned some medium density kind of condos right next to it and some – you know, they just did a lot of little things that and I think that in a way you could say well they didn’t try to change everything; they just changed a few little spaces, right?

Eric: But I think that all these metaphors are just metaphors right; ultimately we have to fall back on actual software; when I look at actual software, one thing I see is that when you want to do kind of sophisticated design approaches, domain modeling, you have to have clarity about language; that’s one of the most important things; you have to have – that’s why the ubiquitous language pattern is one of the like three or four core patterns of DDD.

Brian: Ones people remember.

Eric: Yes.

Brian: Yes.

Eric: And it is; it’s really key that you have this language, which has a certain rigor that is not entirely natural to people but they can learn; and so terms mean what they mean and it’s not just the words but the way the words connect; so that you can express things about the domain and basically say things in terms of model, which is chosen to help solve some really important problem which is an important problem to your organization right now.

But words only mean something in a context, but in a big ball of mud kind of system people have improvised things and they’re not so dependent on the particular precise meaning of words or any other such precision; and there’s a kind of hustle-bustle there and things, building your little delicate thing and it just gets knocked apart so you just have to have that barrier; one of the odd things I think by happenstance early small talk projects well that I was on.

Eric: And one of the reasons that we were able to succeed was that small talk didn’t play too well with others; this was a big problem for us, I mean--

Brian: Yes; it was a utopian model culture.

Eric: Yes; but the thing about it was that it meant that any kind of integration with other systems was something we had to build explicitly; there was a wall between us and you didn’t – you know, if that outer part was a big ball of mud, it didn’t flow into us; we were like in a cement fortress of isolation and that I’m not recommending, but saying by chance what we had created was an isolated, I don’t think we realize what we were doing.

Brian: Yes, we both cut our teeth in small talk and I think one of the things that spoiled you growing up in that environment is just the – it was sort of the best apprenticeship on software design you could ever have had was just exploring that space; and one of the things that made it work is that it had grown organically; you know, they lived there the whole time; the notion of habitability came out of that world; and unfortunately it showed you what’s possible and then you get out there in the rest of the world and it’s just not like that.

Brian: You know, one of the things said about I was resonating with was anytime that you’ve tried to impose order from above in a thing that has this organic character, I mean if you can see that software is organic material or mud which was I think one of the themes that came out in our session yesterday; once you’ve acknowledged that then that kind of stuff, you know, biological systems grow in a way that’s self-directed; you know, when you try to be teleological about it, when you try to dictate it, you screw it up; top down just doesn’t work as well as letting the people who live there reconcile those things locally and that’s one of the reasons why Brooklyn is a way nicer place than it was 20 years ago and it’s one of the reasons why code bases a bit a time can get better.

Eric: So the worst, most feted software thing I’ve seen get healthy; see okay I think that, okay, there’s various things here; one is whether this kind of organic metaphor where does it apply because I don’t think it applies to all software issues; like for example this large system view, I think it’s a pretty good metaphor for that; although I think that other ones that I would use might be kind of cultural or linguistic ones where you have kind of pockets of different cultures and languages and those bump up against each other and are quite resilient too; you know, you can conquer a--

Brian: They’re very durable.

Eric: -- country and yet go back a hundred years and you can barely tell that the conquest had happened right?

Brian: Sure.

Eric: So it’s not like every place that gets conquered the culture just flips like a switch.

Brian: That’s amazing how persistent the Brooklyn accent has been.

Eric: For example and so there you are in a software system and you have all these efforts to transform it and nonetheless there’s this kind of persistence to the way things happen in certain subsystems; and I think it’s very related to that, you know. So on the other hand so those kind of -- a different metaphor suggests a different thing that is since in modeling language is so important to me and the crystal clarity of concepts and concepts are carried by cultures; so you have to kind of carve out a little pocket in there where your new culture can grow and your new language can grow, but in order to solve some problem that really needs that; so I don’t want to do that for its own sake because it’s really hard; but I want to do it for something that is like strategically important or something that is so intricate that no one could do it otherwise; and then I don’t fully expect it to stay like that forever either; I mean there is a naturalness to that; people just don’t keep that degree of rigor naturally; so I expect that eventually it will become more fluid but by then the thing will have been done that nearly impossible thing will be there.

You know, to extend the mud metaphor, there are certain things you couldn’t create in a big ball of mud, but once you’ve created them if you let that mud ooze in around them it has a kind of almost preservative effect; you know, it’s like the La Brea tar pit kind of mud eventually.

And so features never disappear from a big ball of mud; but to actually answer your question, I must admit that for the most part, although I have seen a few messy systems replaced or improved, for the most part the successes I’ve seen are more around the creation of a bubble in which something really important gets them in a finite period of time and not an attempt to change the way the long term structure of the system is so much but more of the temporary but strategically chosen bubble but not always; I mean I don’t say that’s the one answer for now and always but I do think that’s mostly where I’ve seen like success.

Brian: You can build forts in the swamp or enclaves, places where things are locally better and you’re surrounded by trackless mucky waste and you kind of deal with it through things like what you’re talking about yesterday and any corruption layers.

Eric: Yes.

Brian: You know, it’s a phrase I don’t want to use across the street from the United States Court house in Brooklyn; you know, we’ll scare all the politicians but it’s an evocative name; you know, how would those kinds of things connect those little settlements in this swamp that you’re readily gentrifying.

Eric: Right; so if total isolation were and we wouldn’t need anticorruption layer so --

Brian: You’ll be back in small talk.

Eric: Yes; you just create piece of software that’s completely removed from everything else and then you basically have a green field, which is I think that sort of the space in which a lot of these design approaches are imagined is in green field but of course we don’t have very many green field anymore; but the trouble is you can’t do it in total isolation, you have to integrate; so how do you that integration; what tends to happen is well they have some data over there that you need so you bring it in and you start manipulating it and as soon as you do that data, data is not neutral, data is always structured in some way around some concept; the labels of the data have names that the carry the outside model in and so forth so you have allowed, you know, you’ve brought that force into your city and there’s something in it.

So instead of that, you translate, you translate at the border where it comes in; you say well in my context within that conceptual system in my context, this piece of data means this and you just take the whole thing and you translate it as you bring it in; and now the reason they call it an anticorruption layer is because there are situations where you might have a bridge of sorts that would translate between those two things that we built together etc.; an anticorruption layer suggests that I’m keeping you out, you’re not really helping so I’ve built this thing; and from your point of view, you’re just passing me data maybe just--

But I’m taking it apart, translating it, using it; if I’m going to answer a question from you, I’m going to translate it back through this; I’m not going to – my system won’t figure out how to say it in your language but the anticorruption layer will; so now I’m in here and I can pretend that I’m in that isolated world; and any time I need more information, I have to go beef up the anticorruption layer; it’s not a panacea; the anticorruption layer is an expensive piece of software in its own right, but it’s what makes it possible to actually work in that clean little world and that’s necessary if you want to apply most of these kind of high design concept.

I could imagine a whole different approach to software patterns that really acknowledge the ball of mud and worked within it.

But things like separation of concerns and you know reduction, light coupling and etc., etc., all these things we don’t – you know, dry principles and lack of duplication, etc., etc., these things wouldn’t be part of that; that’s not endemic to a ball of mud.

Brian: It’s good.

Eric: I have to let go of all of that.

Brian: Instead of describing, reminding – there’s a Jim Carey movie a few years ago the Truman Show where he lived in this totally artificial world that was made to look like the real world because there are all these facades around like movie sets where he interacted with the reality that he wasn’t it, you know; the way I kind of imagined your local bubbles with anticorruption is that you’ve got these facades that make the outside world as ugly as it is look like what these things would look like in an ideal world that you had built yourself; that always seem to be one of the best uses of the old gang of four façade pattern then you’ve got mediators out in the mud; you know, they are the glue; they’re somehow delivering this stuff up to your border checkpoints and --

Eric: Yes.

Brian: At that point, the world looks like this little enclave you built would in an ideal kind of space; and that’s what that local language can come in; you know, it’s the notion that within that space there’s an internal coherence to the metaphors that you’re using and in the next bubble out there, you know, this alternate universe that we can’t see, they may be speaking a totally different language; and somehow there’s a commerce in these swamp buggies out in the mud that delivers these, the nuggets of actual data between these different enclaves; you know, so it’s a --

Eric: I would say that’s a good description and as you – you know, façade and mediator and so forth, those are good patterns; they’re more kind of described the software whereas --anticorruption layer is meant to describe that.

Brian: …conceptually.

Eric: Yes, the intent of it and its relationship between these two systems; so when you’re actually building that thing, you might well apply those --

Brian: That’s one of the things I always liked about your style this embrace of metaphor because, you know, basically there’s nothing actually going on in the computer except that you have this carefully choreographed electrical thunderstorm that then looks metaphorically like 1’s and 0’s and then looks like something that was elicited by this thing that we call the program; I mean none of these ideas we were talking about, none of the names you’ll talk about in DDD, they’re not actually real at runtime, you know.

And we’re introducing all the super structure metaphor so we can cope with the artifact; the artifact seems real to us; I mean metaphorical places, there’s lots of metaphorical places that are utterly tangible to us and a working code base is one of those; that place you are inhabiting in the old small talk days, those are real place, you know, like you could navigate it, you knew where you were, you knew where other stuff was, you knew where you had to go to get something, you knew what tools to use to find them; it was a very palpable sense of place; and I think, you know, what we both talked about is trying to converge at is day to day that old Dick Gabriel notion of habitability comes into play; you want this artifact to be a place where you can at least get around well enough in the spots where you’re actually doing something useful, yes, to not be up to – right here in mud all day; and that involves, you know, that entails getting the conceptual integrity of these names, these ideas straight and it entails using them intelligently in the artifact in such a way that you’re useful.

Eric: That’s and interesting question; so one thing is that we always and I mean always drive things from a concrete scenario, a concrete example and I say that because a lot of times people don’t; and so they’ll start from sort of visualizing the domain, you know, what are the fundamental concepts in this domain and they’ll start making UML class diagrams and whatever; and this is all fine in a kind of metaphysical exploratory way, but the truth is, you know, we just don’t care about that stuff; we care about what is the problem you’re trying to solve so we start at that; and as a consequence though, one of the first things that tends to happen is that a lot of the domain model concepts that people had built up just turn out not to be that relevant and we get focused in on something that is a real business problem.

And so something that, you know, one tendency that people have when they’re doing modeling usually object modeling although we lately have been doing DDD with some non-object systems, but is that they will just tend to focus on a happy path; and by that I mean there will be a kind of an ideal sequence of events in the business and the model will reflect that and then the other cases will sort of be special cases around that; and that’s almost the reverse of what you get when you start to look at these scenarios in the way that we do; because what the business people are usually worried about well one way of looking at it is if things really happened that way in the business most of the time, what would they need the software for; things will just run like just like on smooth rail.

So what they need the software for is to help them when things don’t go in that exact way; so you start focusing on the problem cases and you start creating a model that actually makes the problem cases clear and simplifies your understanding of those problem cases and brings order to those; and the happy path could conceivably become a little more complex or it could become less natural to express but who cares; that’s really not the case that worries anybody, it’s all those problems.

So you get a model that’s more focused on problem cases and that it more reflects the kind of things that worry the business people and can really express the sort of unhappy scenarios and show how they get resolved; and I think that that probably is the thing that surprises people most is they look at their model afterward, you know, where the project is kind of getting underway and building stuff and they say, this is like so different from what we started with because it’s focused on a whole different area.

Eric: Yes; so I’ve always said I really don’t think models are a thing that – you know, you can’t touch a model; a model is just a system of abstractions; where you --

Brian: Tell that to Tom Brady.

Eric: Yes, well.

Brian: No.

Eric: So of course there are many meanings of the word model.

Brian: Right, yes.

Eric: When we talk about models in DDD, we’re talking about this really just a system of abstractions, concepts and relationships between concepts and so but it does manifest in various ways; and like a UML class diagram is a manifestation; it’s an attempt to describe certain structural characteristics of a model; that is to say well this model contains the concept of these and these and they have these relationships; but that’s a pretty limited view of most of the models that we do; it doesn’t give you much perception of the action and even like the, you know, object interaction diagram shows more than mechanics, it’s not showing what the concept of that interaction is; so I do think it’s important to avoid trying to turn it into a tangible thing; where I think it -

Eric: I think that UML can be useful if you remember that it’s just an attempt to describe certain things.

A more fluid and I think more effective one ultimately is the language we speak; when we talk about how our system works, you can get a level of nuance that is hard to convey; an example that I use a lot is if you’re working on a shipping system and let’s say you have an itinerary and you want to change it; you say well we’re going to use this itinerary up to this point and then after that we’re going to change it; so in English you might say something like truncate the itinerary at this point and then extend it with this other itinerary something like that; you could say that in English and you could actually say that in Java code too more or less; but try to say that in UML; I mean it’s hard to do that in UML; I could draw a picture that would convey it very clearly to you but it wouldn’t be a UML picture; it would be some kind of a conceptualization of itineraries as a picture; and that I think is fine.

And so I think UML can be very limiting and gets people focused on the wrong thing but in language probably it’s just as limiting I don’t know but it’s the thing built into us; so why have this other thing instead of the thing that’s built into us, which actually maps very well in certain ways to programming anyway; and then I really do insist on the program reading like the way we would describe it; this is part of the ubiquity of the ubiquitous language; like if you’re looking at test cases, if there is some test for this truncation function, you know, it should read like well here I’ve got an itinerary and then I say truncate the itinerary at this place; the code given of course that syntax is quite different.

But nonetheless, it should be recognizable as that same statement and this has been one of those basic principles that I think came out of – I’m sure I picked this up in a small talk culture; it was very much let’s create a little language in which we can express our domain problem and then the actual application part is relatively simple; but backing it up is all of these classes and so forth and methods that allow you to basically have that language at your disposal; so the most tangible part would be the code and the least – and I think one of the most important though is the way that people talk.

Brian: Well I’m glad to hear that one of you, one of the things you emphasized is that this, the ubiquitous language flows down into the code; because one of the things that concern; when UML first started getting big, you know, back during the ‘90s, I think one of the potential crimes against computer science that was inflicted on the community or again the programming community was that it exacerbated this estrangement between people who thought they were real architects and the coders; there came to be this view; you know, I used to kind of cruelly say they became a flaccid cabal of cartoon worshippers drooling over UML center folds; they became totally divorced from the code.

They thought gee, what a great gig, we’ll just draw these cool pictures of the system and then we’ll heave them over a wall to people we call coders whose salaries we will reduce and what they’ll wind up getting as a result of doing something like that is get it over balls of mud, in each of those things that they’re building; what’s more when you create UML as a programming language and try to do top down design instead of letting organic growth occur, the position you put those poor doggies in the trenches into is that they’ve then got to correct all the mistakes that these high minded, you know, but relatively shallow analyses of the overall system have inflicted on them by going around, you know, by finding their own paths through the jungle that completely in reality undermine this pretty picture of this system--

Eric: Yes.

Brian: --that never actually got to exist because the guys who had to make it happen couldn’t build it that way because it was ridiculous, you know.

Eric: Yes; in fact I mean that’s beautifully put; I made a very similar point in my book; I mean part of the points is ubiquitous language problem is that if the concepts that you’re capturing and all these design or whatever things, if they’re not reflected in the code they’re simply irrelevant; they are either irrelevant or they’re a downright distraction; a conception that it actually throws people off and describes some fictional system that is not the one.

Brian: True.

Eric: And also at the same time, takes away the tools that people have to really address the domain; so the importance of iteratively refining the code to reflect a model which is our answer to the problem, if you take this away from people by imposing some kind of UML diagram that they have to adhere to, you’ve taken away their ability to really use models to address their problem and now they have to address it another way and the fallback is the ball of mud.

And so it is amazing to me how persistent this notion has been over the years that modeling is about drawing UML diagrams; that anyone who talks about large systems must be an architect and anyone who – and must also be removed from the code; all of these things are things which have been – I mean I know they’re very common but I’ve been frustrated by those tendencies for most of my career, I think probably because I like to write code and so this desire to make code go away always mystified me; why is that even a goal; isn’t that like the most fun part of our job anyway. So why is it that we want to draw UML diagrams which is not very fun instead of write, you know, small talk code, which is fun; it just baffles me you know.

Brian: You know, one has to fall back on a Dickensian tradition, description of the state of the software industry, best of times, worst of times; you know, it’s a golden age for mud, it’s a lahar of mud out there; you know, we are producing incomprehensible code at an astonishing rate, you know, probably way quicker than our logistics chains are ever going to let us maintain it; and yet there are things out there that are allowing, pockets of sanity, pockets of gentrification and runs around, you know, horrible, existing things; you know, it’s alarming, it’s exhilarating, it’s an interesting time to be in this racket. Stay at it guys!

Eric: Yes; okay, so I’ll react to that I think we went through a really bad time in the late ‘90s and the early 2000s in terms of the frameworks that were imposed upon us just made the kind of things I liked to do nearly impossible. I mean when I look back on the way it was to be on a software project where they had said you must use J2EE with all of its entity glory, just it was awful, and the OR mapping which we still use a lot of but which we’re beginning to see, you know, alternatives pop up with the No SQL; and so finally I think we’re coming out of a tunnel of these terribly heavy, overcomplicated frameworks that impose so much overhead that actually doing the kind of iterative development that reflects models and so forth that I think is the key to success with domain modeling was just incredibly hard and not much fun; and now there’s all kinds of stuff popping out and I wouldn’t try to predict which thing is going to succeed and which ones will fail; I think we’re going through one of those tremendous experimental periods and so as you say it’s exhilarating and I’m glad it’s happening.