Technical – 345 Systemshttps://www.345.systems
we help you build amazing softwareWed, 20 Feb 2019 15:47:16 +0000en-UShourly1https://wordpress.org/?v=5.0.3What’s the Future for OO Programming Languages?https://www.345.systems/architecture/whats-future-oo-programming-languages/
https://www.345.systems/architecture/whats-future-oo-programming-languages/#commentsSun, 07 May 2017 23:00:56 +0000https://www.345.systems/?p=625I spent most of my software career either writing, designing or architecting solutions built with a heavy object-oriented (OO) bias. I simply cannot even estimate the number of lines of C# that I’ve written or read. Even now, I love the abstract elegance of good OO code. However, I’m left wondering how much of a...

]]>I spent most of my software career either writing, designing or architecting solutions built with a heavy object-oriented (OO) bias. I simply cannot even estimate the number of lines of C# that I’ve written or read. Even now, I love the abstract elegance of good OO code.

However, I’m left wondering how much of a future it has left.

Here are a few thoughts about where mainstream programming is going, and what choices programmers will favour.

Note: When I talk of OO in this article I’m referring to strict, compiled OO languages such as C# and Java, not to dynamic languages that support objects and classes like Python and JavaScript.

Web APIs

Microservices, the second and more complete incarnation of Service-Oriented Architecture (SOA), allow us to decompose our logic into reusable services that are accessed via open network protocols. This is crystallising into an accepted convention of REST/HTTP. The code for our microservice, i.e. what’s behind the API, doesn’t matter to the outside world. Is it OO? Is it even compiled? Who cares?

Indeed, programming web APIs is functional programming in its very nature.

This is partly why JavaScript is such a good fit for server-side code. Whilst the language has been updated to incorporate many OO features such as classes, the language is functional at its core. To define a REST API in JavaScript is really just to map a function onto a route. And since functions are first-class citizens in JavaScript, well that’s just dandy.

By contrast, take a look at some of the implementations of service interfaces in OO languages (e.g. .svc and .asmx in C#) and you’ll notice that they have shoehorned in an OO interface / class structure when in fact an API call invokes a function. You find that to even create a “Hello World” you end up with a binary of an amazing size, because of all the plumbing that’s being done for you under the bonnet.

Look at the size of a NodeJS JavaScript “Hello World” app. A kilobyte? It’s faster too. A lot faster. Functional programming and functional languages definitely have the edge in this area of programming.

Serverless Functions

I don’t know how many of you have used AWS Lambda, or its equivalents (Google Cloud Functions, Azure Cloud Functions). This is a nascent area for programming, which is coming into being because of cloud platforms. Essentially, serverless functions are pieces of code that are invoked by an event. However, they don’t have a permanent host hence the “serverless” moniker.

When an event trigger occurs, such as a message being placed on a queue, the cloud platform spins up an instance to run the code that handles the event. When it’s finished, the instance can terminate and hence free up resources. This is an amazingly powerful concept, because it really is getting as close as possible to a pay-as-you-use model.

And the programming paradigm? Functional. Dynamic. You invoke a function to handle the event and the data that comes with it. Ideal language for this type of scenario? Python.

What you can achieve with a few imports and a few lines of code in Python requires a mind-bending amount of references and components in many formal OO languages.

UI

User interface (UI) is a very interesting area from a technology point of view. My old boss always used to say that the only thing a user ever knew of your software was the screen in front of him. You’d better make that good or your users will all assume everything else is rubbish as well. No matter how unfair, that’s the way it is.

As a result, UI is the area where the real pace of change is felt. You only need to go back a couple of versions in many pieces of software (or websites) and you will cringe at the clunky interface you used to take for granted. UI sells.

Good UIs are stateful too, and hence they lend themselves to OO design. Browser-based UIs were something of an exception, as postbacks to the server to refresh HTML content were effectively functional web API calls. The trend towards single page applications and frameworks such as ReactJS and Angular have meant that even browser-based UIs are now falling in line with OO programming methods. Ironically, using JavaScript.

I see UI as one of the last great outposts of OO programming.

Frameworks

The open source movement, combined with open package repos and package managers (e.g. NPM, pypi, NuGet) have massively impacted on the way we use code. Software is built on software, and if a problem has been solved before, well you’d be insane not to reuse the solution.

In terms of language and programming paradigm the jury is out on this one. There’s such a split in the packages that are out there that it’s hard to make predictions. There is a ton of functional JavaScript code out there, just as almost everything on NuGet is probably OO.

This is an interesting one to watch. Frameworks and framework extensibility were a case study for OO design patterns. Now, the functional world has found its own patterns for extensibility and reuse. I’m interested to see what wins out in this area.

Other Trends

The trend away from installed on-premise software to Software-as-a-Service (SaaS) is only going to accelerate. This will build on the points above, but the real demise will be felt when companies are decommissioning system after system from their own datacenters to avoid having to maintain and support them.

Add to this the shift to cloud. Do the math – cloud represents huge savings and huge opportunities if embraced correctly. Less so for pure lift and shift of on-premise systems, but organizations that are architected from the ground-up to take advantage of cloud and SaaS will have huge competitive advantage. Again, this is more of the same: APIs, Serverless Functions, UI.

DevOps is a hot topic, and one close to 345’s hearts. If you look at what’s going on there you find a lot of scripting going on. Python’s very strong, but so is Ruby. Bash is very procedural, PowerShell mostly procedural but also made to support .Net. I think Python is going to win big in this area over the next few years. Just look at the AWS CLI. Python is also widely adopted within Google.

Summary

This has been a whirlwind tour of a big subject area, sprinkled with a thin layer of thoughts. By all means add your own input to the comments.

I think the big growth areas in programming are going to be (1) further decomposition into microservices and APIs, (2) serverless functions to handle asynchronous processing, (3) SaaS applications replacing installed applications.

Of these, I think the first two will be dominated by functional programming and dynamic languages.

If you’ve been used to OO you’ve probably told yourself time and again that you’re doing it the “proper” way and those functional programmers / scripters are cowboys. Free your mind to accept the fluidity of a functional language: the possibilities are huge. And realise that the scripters aren’t cowboys after all, they just have a different way of thinking. They don’t rely on compilers, they rely on tests. Which is a good thing.

]]>https://www.345.systems/architecture/whats-future-oo-programming-languages/feed/2Python: The Dirtiest Language Everhttps://www.345.systems/technical/python-dirtiest-language-ever/
https://www.345.systems/technical/python-dirtiest-language-ever/#respondMon, 20 Feb 2017 00:00:37 +0000https://www.345.systems/?p=377This week I wrote Python for the first time ever, and boy is it dirty. Every time I write some code in it I feel unclean. I spent many years doing OO programming in C#; I have been used to JavaScript, and whilst JavaScript is untyped and interpreted there are so many fine points in the...

]]>This week I wrote Python for the first time ever, and boy is it dirty. Every time I write some code in it I feel unclean.

I spent many years doing OO programming in C#; I have been used to JavaScript, and whilst JavaScript is untyped and interpreted there are so many fine points in the way the language works that it offers patterns and solutions that are virtually unmatched anywhere else.

But Python. Oh my. What a bad language you’ve been. The way default values magically morph into global variables, the way that you can’t seem to control or deduce the scope of variable declarations and assignments. The documentation for interfaces has been sketchy at best for much of the time, possibly because there is not really any such thing as an interface in the way I understand it. It’s been a culture shock, to say the least.

My kids have been learning Python in school, and even they had a bad time with it. The school was teaching Python because it’s easy, but to me that seems wrong. The first language I was taught – as opposed to self-taught – was Pascal, a language designed to be typed, structured, compiled and precise. Learning in a language that has a lot of disciplines is a good thing, because it teaches good habits that stay with you.

Nevertheless, I have been persevering. You may be wondering why.

Despite my earlier rant, not everything is bad in Python. The real sweet spot for Python seems to be (for me) in command line scripting. When you feel that bash has run out of steam you can flip over to Python and work with something that’s richer. The aspects of Python that appeal in this scenario are (1) the ability to maintain complex state in variables and (2) the libraries and modules you can build on.

Let’s take variables for a start. If you’re scripting in bash you can always declare variables, or call on environment variables, but they are all pretty one-dimensional. In Python I can actually declare structured types via a JSON-like syntax and then tag on properties and arrays as needed. This is a massive improvement on bash.

The second is the modules. I have been migrating a command line tool with quite a few options and parameters. In bash the parameter parsing and help text took up about 300 lines of code in the end. In Python, using the click module, I have stripped nearly all of this out by using the built-in decorators. And made the tool work better as a result. And improved the help text. And improved the way output is printed to the console. And improved the error handling. And improved the testability. And…and…and…

Want to output your code as JSON? json.dumps()

Want to load a JSON response from a service? json.loads()

Make a PUT request to a REST API? requests.put()

At the start of the exercise I was unsure where I’d get to, or if I’d even get anywhere. In the end I covered so much ground in a short space of time I couldn’t believe it. I find that when I’m working as a programmer the feeling of productivity, of delivering great value in short time, is one of the most fuzzily rewarding feelings I get. I got it this week in spades.

]]>https://www.345.systems/technical/python-dirtiest-language-ever/feed/0Best practices for tracing and debugging microserviceshttps://www.345.systems/technical/best-practices-tracing-debugging-microservices/
https://www.345.systems/technical/best-practices-tracing-debugging-microservices/#respondMon, 13 Feb 2017 14:00:58 +0000https://www.345.systems/?p=358In this article I’ll be talking about some techniques for debugging and fault finding in microservices architectures. One of the issues we’ve had for a long time, in fact ever since distributed computing became a thing, is debugging issues with a single business process that is run across multiple machines at different times. When software...

]]>In this article I’ll be talking about some techniques for debugging and fault finding in microservices architectures.

One of the issues we’ve had for a long time, in fact ever since distributed computing became a thing, is debugging issues with a single business process that is run across multiple machines at different times. When software was monolithic we always had access to the full execution stack trace, and we knew the machine that we were running on. When we encountered an error we could write all the information we needed to a log and later inspect it to see what went wrong.

When applications started running over small farms we no longer knew exactly which machine we were running on, but a look over the whole farm was a highly inefficient but often-taken route. With the advent of cloud computing, where compute instances are ephemeral, we cannot rely on machine logs to be there when we come to analysing log data. We need to get serious about how we manage our logging so that we have a decent chance of tracking down and fixing issues.

So, here are my best practices for tracing and debugging your microservices:

1. Externalise and centralise the storage of your logs

The first thing you need to do is to treat the data integrity of your logs seriously. You cannot rely on retrieving information from a physical machine at a later date, especially in an auto-scaled virtual environment. You wouldn’t allow a design where important customer data was stored on a single virtual machine with no backup copy, so you shouldn’t do this for logs.

There’s no absolute right answer to how you store your log information, whether it’s in a database, on disk or in an S3 bucket, but you need to be sure the storage used is durable and available.

Tip: If you’re on AWS you can always direct your output to CloudWatch. On Azure consider using Application Insights. We will be going into more detail on logging in future articles, as getting this right is fundamental to success in making operationally supportable services.

2. Log structured data

Many of the logging components we get now will output JSON documents for each log instead of outputting flat rows in a text file. This type of data structure is important because log processing and collation tools can easily process each record, and the information you provide in each record can be richer.

3. Create and pass a correlation identifier through all of your requests

When you receive an initial request that kicks off some processing you need to create an identifier that you can trace all the way through from the initial request and through all subsequent processing. If you call out to other components or microservices in order to complete your processing then you should pass the same correlation identifier each time.

Each of your subsequent components and microservices need to use this identifier in their own logging, so you can collate a complete history of all of the work done to process a request.

4. Return your identifier back to your client

It’s all very interesting if you can trace the flow through your system for a given request, but not much fun if you get a support call and you have no idea which request you need to be looking at.

When your API client has made a request, and initiated a process, you should return a transaction reference in the response headers. When your ops team needs to investigate the issue they should be able to provide the transaction reference with a support call and you can find the relevant information in the logs.

5. Make your logs searchable

It’s a good start to capture your log data, but you’re still left with the haystack of information even if you have the identifier of your needle. If you’re going to be serious about support then you should be able to search and retrieve a collated and filtered set of information for a single request.

Even if your data isn’t initially being written into a data warehouse you might want to look at offline processing that loads it into one. You’ve got plenty of options for this, such as the ELK stack or AWS Redshift. If you’re on the Azure stack you could choose Application Insights, as mentioned above.

6. Allow your logging level to be changed dynamically

Most logging frameworks support multiple levels of details. Typically you’ll have error, warning, info, debug and verbose as the default logging levels available to you, and your code should be instrumented appropriately. In production you’ll probably log info and above, but if you have problems in specific components then you should be able to change the tracing to debug or verbose in order to capture the required diagnostic information.

If you’re getting problems you should be able to make this change to logging levels on the fly while your systems are running, diagnose the issue, then return the logging back to normal afterwards.

Summary

Fault finding in distributed microservices can be difficult if you don’t readily have access to the logs from all of the machines your code runs on. Virtualized cloud instances, that can disappear at any time, exacerbate the problem because you cannot go back to a machine instance later to see what has happened.

Planning the management of logs and how you conduct fault finding needs to be done at the design phase and executed with the appropriate tooling and technique. Once you’ve cracked this, supporting your systems will be a whole lot easier.

]]>https://www.345.systems/technical/best-practices-tracing-debugging-microservices/feed/0The anatomy of a release pipelinehttps://www.345.systems/technical/anatomy-release-pipeline/
https://www.345.systems/technical/anatomy-release-pipeline/#respondMon, 30 Jan 2017 00:00:55 +0000https://www.345.systems/?p=341Following on from my previous articles on DevOps I’ve decided to write in more detail about the release pipeline. DevOps is a such a buzzword at the moment, but under the bonnet what is actually involved? In this article I’ll be dissecting release pipelines, which transform Continuous Deployment from an aspiration to a reality. I’ll...

]]>Following on from my previous articles on DevOps I’ve decided to write in more detail about the release pipeline. DevOps is a such a buzzword at the moment, but under the bonnet what is actually involved?

In this article I’ll be dissecting release pipelines, which transform Continuous Deployment from an aspiration to a reality. I’ll talk about the underlying building blocks that any good release pipeline will be comprised of and how they fit together.

CI versus CD

I think of Continuous Integration (CI) as being distinct and complimentary to Continuous Deployment (CD). I think of CI as a check on code quality and not of system quality. CI includes the following operations:

Static code analysis

Semantic code analysis

Compilation

Code packaging

Unit testing

CI should run on every branch you push to measure and ensure quality throughout your entire codebase. CI should test every pull request before you merge into a protected branch. Successfully passing CI is a prerequisite for your code ever seeing the light of day.

Continuous Deployment is different. CD is the automated (or semi-automated) process by which committed code is released to production. CD is usually triggered by commits to a single branch (e.g. master) or a small subset of branches. You can safely commit new features into any old branch you like, safe in the knowledge that you’re not going to put it live unless and until you then merge those commits into your CD branches. After that, you assume it’s going live.

Release Pipeline

The way we achieve this is to put into place a release pipeline. A release pipeline is the conceptual process by which we take committed code into production. As such, a release pipeline can be as ephemeral or as real as we want to make it.

The fundamental release pipeline, code change to production software.

There are some underlying capabilities we need in order to make sure we actually do have a release pipeline though:

A means of triggering the pipeline to run.

A means of executing tasks such as environment provisioning, application deployment, testing, collation of test results.

A means of controlling the flow of execution, especially to stop further processing in the event of failure.

An artefact store to maintain state throughout the process.

A metadata store, allowing consistent metadata (such as build number) to be passed to each stage of the pipeline.

A configuration store, allowing environment-specific values to be retrieved for use in the pipeline.

Logging of the work performed and of any errors.

Notifications of success and failure.

A release pipeline can also be thought of as a workflow. A workflow whose purpose is releasing software. As such, modelling your release pipeline should simply be a case of modelling how you want to release your software.

Triggers

Releases happen in response to one or more events. Often the triggering event is a code commit, but sometimes the release can be triggered manually or on a schedule.

You may also want your pipeline to run automatically up to a certain point (e.g. to the completion of pre-production testing), and then require manual approval to actually release into production. You may therefore want manual triggers to act as prerequisites for completion even though the start of the pipeline process is run automatically.

Pipeline Stages

Pipeline stages are the control points in your release pipeline, and within a stage you have tasks that are triggered. The key things to note about pipeline stages are:

A stage cannot start unless all of its prerequisites are fulfilled.

A stage cannot complete unless all of the tasks within it are complete.

Failures of any task (usually) result in the whole stage failing, and in turn this usually fails the whole release.

Basic release pipeline stages, a sequential series of steps.

A simple pipeline like the one pictured above is a sequential series of stages. These types of release pipeline have usually descended from CI systems, where a build process is executed linearly from start to finish.

A more complex pipeline may use fan-out and fan-in, as a means of running stages in parallel and then collating the output from each. This can be an advantage in highly complex deployments where there are a series of services that need to be deployed and tested as part of the overall process.

A pipeline supporting parallel processing via fan-out and fan-in.

The control of flow in your pipeline will vary depending on which model you use. In a simple pipeline you can trigger the start of the next stage from the completion of the previous stage. In a fan-out fan-in model you need to find a way of allowing stages to subscribe to their prerequisites. I’m not talking about underlying tech in this article, just what needs to happen.

Repair and Restart

An optional feature in release pipelines is the ability to repair and restart. This can involve correcting environmental issues and then allowing failed tasks to start again and then hopefully resume the process.

Repair can work both ways though. If you find that you’re fixing machine settings then you really ought to be working on fixes to your infrastructure provisioning scripts instead of masking your underlying issues. If you’ve has a network blip then restart may be fair enough.

Not all release pipelines support repair and restart, and if yours does then use it wisely.

Tasks

Tasks are the things that actually get done, at a granular level. Importantly, within a stage it should not matter which order the tasks complete in. Use stages as the gates in the control flow and tasks as the things that actually get done.

The tasks you will need include:

Infrastructure provisioning. This can include spinning up new virtual environments for test, or it can involve ensuring that a test environment is configured correctly and that the required services (e.g. web server) are installed and running.

Application deployment. This includes taking the packaged software and deploying onto the infrastructure instances, and making any environment-specific configuration changes as required.

Testing. Executing tests and publishing test results. You also need to be able to mark a stage as failed if the test run is not successful.

Infrastructure shutdown. After running a test phase any virtual infrastructure can be shut down or even decommissioned entirely to save costs.

Some tasks will be asynchronous, and your pipeline may need to be able to handle this. For example, and application that spins up AWS EC2 instances may have to wait a minute between the start of infrastructure provisioning and application deployment or testing while the environment is prepared.

Artefact Store

In general, a release process starts with a code change and ends with provisioned infrastructure and deployed software. Along the way the packaged software needs to be available for deployment to each environment, and may need to be modified and / or configured for each environment. An artefact store therefore underpins the process.

A release pipeline with a supporting artefact store.

The artefact store also needs to support distinct artefact versions. The set of artefacts for a single build and release should be atomic and distinct from the artefacts from any other release with no cross-contamination.

Going back a few years we used to achieve this by having a network share for the build and release process, and each build having its own folder within that share. Whether you take this approach, whether you use a database, or even put everything into an S3 bucket, persistence of artefacts is essential.

Configuration Store

The artefact store contains different data (software) for each build and release. The configuration store contains values that are consistent between builds such as connection strings, API URLs, environment-specific users and permissions.

A release pipeline with a configuration store.

The configuration store will ultimately contain some of your production configuration, even if it’s just machine names, and so it is essential that your configuration store is secure and encrypted. Your release pipeline should be able to pull out the required configuration for any environment at the relevant stage of the pipeline and use it to allow you to provision and deploy.

Logging

It goes without saying that if there are problems with your pipeline execution you need to be able to examine your logs to see where any problems occurred and what went wrong. In a system that has a lot of moving parts you need to ensure that the logs are collated in such a way that you can make sense of them.

A release pipeline supported by a log store

Everything you log will therefore need to be stamped with at least the following information:

Release pipeline / application

Pipeline stage

Build number

Timestamp

Beyond this, it’s up to you how you visualize your logs. A very basic system might just collate them together and make them searchable, but sophisticated build pipeline software will give you a graphical view of the execution of your pipeline with drill-down into the logs.

Metadata Store

The metadata store is one of the simplest features of the process. It is usually a collection of name-value pairs that contain build-specific information. Often these are exposed as environment variables, but depending on how you conceptualise your build data this could also be information on completion of stages and tasks.

A release pipeline supported by a metadata store.

Execution Engine

The release pipeline involves executing a workflow. How this actually works under the hood depends on implementation, but you need processes to execute. Whether this is a glorified bash script or a hosted workflow engine, somewhere the work actually needs to get done.

A release pipeline powered by an execution engine.

Notification Service

Notifications are optional, but useful and almost universal. When your process succeeds or fails you generally want to tell someone about it, and usually that’s via email. If you’re in the modern SaaS webhooky world (like 345) you might prefer to use a tool like Slack for notifications. Whatever your preference, someone needs to know.

A release pipeline with a notification service.

Process Viewer

Optional, but useful, is a graphical view of your pipeline. It’s no coincidence that CD tools include some sort of graphical view, because it’s a truism of software that UI is the only thing people see of their software. UI is the tip of the iceberg, often the smallest part in terms of code and functionality but the only bit you can see.

A release pipeline with a graphical process view.

Summary

So that’s the end of my tour of a release pipeline. I haven’t talked about specific tools and technologies because I wanted to keep it at a conceptual level. I’ve highlighted the main features that allow the execution of your release process. Whether you’re rolling your own release pipeline or using a package, you should be able to spot the same feature come up again and again. They may take different nomenclature depending on how the developers’ vocabulary evolved and the paradigms they were using, but the essence will all be there.

]]>https://www.345.systems/technical/anatomy-release-pipeline/feed/0TypeScript – Help or Hindrance?https://www.345.systems/technical/typescript-help-hindrance/
https://www.345.systems/technical/typescript-help-hindrance/#respondMon, 26 Dec 2016 00:00:23 +0000https://www.345.systems/?p=283At 345 we’ve been using TypeScript for a while now. This started when we began using Angular 2 during the early Betas, and we’ve been experimenting with TypeScript on some of our other Node.js applications as well. The time has come to ask the question: Is TypeScript a help or a hindrance? What is TypeScript?...

]]>At 345 we’ve been using TypeScript for a while now. This started when we began using Angular 2 during the early Betas, and we’ve been experimenting with TypeScript on some of our other Node.js applications as well.

The time has come to ask the question: Is TypeScript a help or a hindrance?

What is TypeScript?

I’m not going to go into too much detail here, you can always look at https://www.typescriptlang.org/, but it’s worth noting that TypeScript is a superset of JavaScript, which compiles to JavaScript (and you can instruct the compiler on which version of JavaScript to compile to).

Why use TypeScript?

JavaScript is an untyped, interpreted language. When you write JavaScript, your code is executed as-is. Owing to the lack of type safety, some people believe that JavaScript code suffers from errors that could be avoided in other compiled languages. Hence TypeScript allows JavaScript developers to write in a language that is more like a traditional OO language, with compile-time checks.

Although TypeScript compiles to JavaScript, variables are actually strongly typed. This allows the compiler to infer the type of a variable and hence check whether it is being used appropriately.

When TypeScript has been good

Angular 2 (https://angular.io/) runs off TypeScript. When you’re creating single page applications (SPAs) with Angular 2 the majority of the code you reference is the Angular 2 framework itself. The Angular model is consistent and well written, and as Angular provide the types for their framework the use of a typed language is natural and effortless.

TypeScript has also been good for asynchronous JavaScript. ES6 supports a promise-based callback model for async programming, and TypeScript allows you to program async methods and have them compile down to a much more verbose JavaScript implementation. With the advent of ES7, this advantage for TypeScript may not last long. There are, also, JavaScript libraries that allow a similar programming model.

The “linting” process, i.e. code checking, has been good in TypeScript. It has been easy to put in place rules that provide uniform-looking code that passes static code checks. Again, not saying tools for JS aren’t out there, but this has been good in TS.

When TypeScript has let me down

We build a lot in Node.js these days, as do many others. We rely heavily on NPM packages to pull in references, but with TypeScript you need to pull in the types for referenced packages in order to use them. Most of the types are out there because people have needed to use them for their own projects, and they create type definitions and upload them to opensource repos.

Types have therefore become a minefield of inconsistency, with the official types often not keeping in step with the versions of the packages they are supposed to reference. Furthermore, there’s a ton of lazy typings with “any” types used all over the place, negating the benefit of type checking in the first place. TypeScript 2 tidies up the type references significantly, and includes the addition of a type search (https://microsoft.github.io/TypeSearch/) and type references, but there are still problems with the availability of type definitions and the inconsistencies therein.

The massive productivity improvements you can get from using Node and NPM are dampened considerably by the TypeScript type restraints. While you could argue that this is a “good thing” because you get additional compile-time checks, you could counter that by saying that you should concentrate on test coverage instead.

Version drift

TypeScript is very much an evolving language, and the ancillary components such as tslint (https://www.npmjs.com/package/tslint) are also moving rapidly. This can cause issues if you’re not pinning your versions and managing the transition from one to the other carefully. This is a configuration management activity that seems to bleed time over and over again if you’re not careful.

Code coverage

Remember that all TS code actually ends up being executed as JavaScript. Therefore any runtime error stack traces, code coverage stats for unit testing, logging or other debug information will refer to the JavaScript and not to the TypeScript. It’s not too much of a stretch to eyeball one against the other, but it’s something you need to be aware of.

Limitations of TypeScript

Unlike true OO languages such as C# or Java, where an object can be properly interrogated to find the interfaces it implements and the superclasses it descends from, TypeScript is much more limited in this regard. You cannot check whether an object implements an interface. Given the purpose of TypeScript, this seems slightly crazy, but it’s true. You can only check whether an object is an instance of a class, because the runtime type checks uses class constructors. There is no type attributes associated with an object.

I’m mystified why the TypeScript designers didn’t take the opportunity to tag metadata onto classes so that it can be interrogated at runtime, but that’s another story.

Loss of JavaScript flexibility

Ironically, one of the strong things about JavaScript is the very type-unsafe flexibility that you can gain. In JavaScript you can simply augment an object with properties and functions at runtime, which means that much of the inheritance you use in OO languages, or use of the decorator pattern, is actually redundant. I can tag properties and methods to an object, and hence dynamically add an interface, without having to do this at compile time. Powerful, and only to be used wisely.

In TypeScript you end up in this blind alley sometimes where you need to inherit a class just to add a new interface, and then you can’t type check your interface later. You end up spending time getting round the limitations of the TypeScript language in order to achieve things that are supported by the language you’re compiling to.

Compilation

Also, note that in JavaScript applications the code you write is the code that is executed. There isn’t a compilation stage. You can go straight from writing code to running tests with no interruption.

TypeScript by definition has the compile stage, and so you then need to sort out your source from the code you ship, and then in the case of NPM packages, you’ll probably end up keeping both in your GitHub repo anyway. Setting up the compile and running a build process is additional activity that you don’t get with JavaScript.

Summary

TypeScript offers some neat things, and is perhaps a way of getting OO developers to feel comfortable with JavaScript applications.

The major cost in almost all software projects is man-hours. Productivity is therefore paramount, and as far as I can see you lose a ton of productivity when you use TypeScript that you don’t seem to gain anywhere else. TS is the “type-safe tax” you have to pay on your project.

Some specific scenarios, such as Angular 2 apps, you may decide are worth it. I would advise anyone making a decision on TS vs JS to look very closely at the benefits you feel you’ll get and to try to measure the productivity gap between the languages.

Don’t use TypeScript because there’s a bandwagon. Use it because you have a reason to. If you don’t have a reason to, stick to JS and concentrate on mature dev practices and test coverage.

]]>https://www.345.systems/technical/typescript-help-hindrance/feed/0Application Configuration for Highly Scaled-out BizTalk Groupshttps://www.345.systems/technical/application-configuration-highly-scaled-biztalk/
https://www.345.systems/technical/application-configuration-highly-scaled-biztalk/#respondWed, 20 Apr 2016 23:00:00 +0000http://345.systems/?p=201Introduction In this blog post I am going to talk about configuring highly scaled out BizTalk applications, so that application configuration data can be managed centrally and shared across multiple nodes in the application. BizTalk Server and Enterprise Single Sign-On Microsoft BizTalk Server is a product we have worked with for well over a decade...

In this blog post I am going to talk about configuring highly scaled out BizTalk applications, so that application configuration data can be managed centrally and shared across multiple nodes in the application.

BizTalk Server and Enterprise Single Sign-On

Microsoft BizTalk Server is a product we have worked with for well over a decade now, and is still a solid product for on-premise integration solutions. The Enterprise Edition of BizTalk can also be scaled up to handle significant workload, and it has a food configuration story in Enterprise Single Sign-On (ENTSSO).

Enterprise Single Sign-On has at its core a database that holds encrypted secrets, and this is the data store that sensitive data used in the configuration of BizTalk send and receive ports is held.

ENTSSO has an additional benefit in that it can be used to store application config for BizTalk solutions as well. Consider the following schematic of a highly scaled-up BizTalk installation, typical of what we have worked with in some of our larger clients:

BizTalk deployment with processing servers and database servers.

Configuring BizTalk Applications

The benefit of using ENTSSO for app config is that each of the nodes in the BizTalk Group all point to the same configuration store, the SSO database (SSODB) that is deployed as part of the BizTalk installation. We can actually hook into this to give us applicati0on configuration that can be used directly in orchestrations and pipelines.

Adding Additional Non-BizTalk Servers

In a large enterprise application we might have other servers that participate in the application. The most common of these are web servers that are hosting web services / web APIs that BizTalk is integrating with and orchestrating, and application servers that are running Windows services that perform background processing for the application.

We might find that these servers also need to consume some of the same config data that the BizTalk application uses. Fortunately, we can also install the SSO client onto these servers as well, so that our additional servers happily consume the configuration data and benefit from the centralised management, consistent programming model and automatic refresh that we have built into the BizTalk group.

If we update the schematic, we might find we have something like this:

Hitting the Wall on Scaling BizTalk

So far so good, we have a highly scaled environment with centralised configuration data. However, BizTalk groups will always run into limits of scale at some point. This will usually manifest itself either as CPU / disk overload in the message box database or CPU / memory overload on the BizTalk servers. When this happens it is very difficult – and very expensive – to achieve further scale, both in terms of architectural change and in hardware.

A better way to achieve scale is to regard a BizTalk Group as a unit of scale. Ideally, architect this in at an early stage in the application. Ensure that work being fed into BizTalk can be load balanced between more than one group, so you can horizontally scale the BizTalk databases, and you can also take a group down for maintenance without leading to loss of service.

Scaled-out BizTalk application with multiple BizTalk groups.

In this scenario you will want to be able to scale the web servers and application servers as well. There is a crunch issue here though: the ENTSSO configuration service that we used in the previous schematic to provide a centralised configuration store for all of our application is tied to the SSODB deployed within a single BizTalk Group. Which SSO database do we connect to? How do we manage the fact that there is no longer a single configuration store, but one per BizTalk Group?

As we scale out through additional groups, and to be able to deploy / retire groups as unit of scale as part of a flexible infrastructure, we need to solve the conundrum of how to configure the applications across our estate.

cloco as a Configuration Store for Distributed Applications

At 345 Systems we’ve been through all of this pain with large-scale banking systems capable of processing tens of millions of payments per day. We’ve hot these very limits in the scale-out of BizTalk, and we have encountered the pain of non-centralised configuration in a mission critical enterprise.

Developers: You need a configuration programming model that can be surfaced easily in BizTalk and other applications, with a deployment framework so that your development lifecycle drives your configuration store.

Administrators: You need to be able to manage, view, update and revert changes to configuration. You need to ensure that configuration is secure from tampering and information disclosure.

Architects: You need reusable building blocks to use as a bedrock for your applications.
cloco, our Cloud Configurator, gives you all of this and more.

Notes on the SSO Programming Model

Above, we discussed the fact that it ispossibleto use ENTSSO as a configuration store for applications. We did not recommend it, nor did we say that the programming model for this configuration would be easy. We did not say that the centralised administration of configuration would have a nice UI or validation tooling.

When we created cloco we envisaged a programming model that is as easy as possible for .Net programmers. We wanted to create a framework where you simply reference our assembly, create a class that defines your configuration data, create a JSON-serialized document containing your configuration, and that’s pretty much it.

We’re really proud of what we’ve managed to achieve with cloco, because we know how much time and effort you can spare by hooking into a configuration framework that’s designed with developers and administrators in mind, rather than hooking into a subsystem that was designed to be an invisible subsystem deployed as part of a product.
We’d love you to try out cloco to see if it can help your applications.