This is a topic that is well known from previous reinvent conferences. It is especially interesting for large enterprise level organisations looking for the right architecture when consuming public cloud services from AWS. You need to watch it if you are planning to run multiple workloads in AWS.

Big Data Architectural Patterns and Best Practices on AWS

If you are into big data applications, you cannot afford to miss this epic talk by Siva Raghupathy who presents the best practices and architectural patterns for big data on AWS. You will find out how to simplify big data processing and how to choose the right AWS services for your application.

Security Anti-Patterns: Mistakes to Avoid

Security is not something you can take lightly when deploying business applications into any public cloud. This talk showcases patterns that are best avoided but also covers DevSecOps best practices. You will also learn how to leverage native AWS services to meet the strictest enterprise security requirements.

How to Design a Multi-Region Active-Active Architecture

This talks explains what you should consider before deciding on an active-active multi region architecture and the trade-off decisions that you have to make on cost, performance and consistency. You will learn what services you could use and how to deliver an active-active multi region architecture on AWS.

Serverless Architectural Patterns and Best Practices

Serverless architecture is becoming more and more popular. This is due to the obvious benefit: no more provisioning and managing of servers. This session showcases reusable serverless patterns and security best practices. It goes well beyond AWS Lambda.

]]>https://pajdzik.com/2018/03/18/the-5-talks-from-aws-reinvent-conference-2017-that-you-should-not-have-missed/feed/0My impressions after taking the AWS Certified Security – Speciality Beta examhttps://pajdzik.com/2018/02/11/my-impressions-after-taking-the-aws-certified-security-speciality-beta-exam/
https://pajdzik.com/2018/02/11/my-impressions-after-taking-the-aws-certified-security-speciality-beta-exam/#commentsSun, 11 Feb 2018 11:33:07 +0000http://pajdzik.com/?p=636AWS have been running the beta version of the AWS Security Speciality exam since the beginning of this year. You are given 50% discount if you decide to take the exam and a free re-take if you do not pass for the first time. The downside is that you may have to wait for the results untill May.

I took the exam on 10.02.2018 in west London and decided to put together a quick overview of my experience and a list of video resources that I would suggest using for preparation:

There were 70 questions to go through in 180 minutes. I had more than enough time to go through all the questions and have almost an hour left to review questions that I wasn’t sure about.

The majority of questions involved IAM, KMS, Cloud Trial, Config and Cloud Watch. You need to have quite detailed understanding of the services to be able to pass the exam.

The questions were not as complex as the questions from the AWS Solutions Architect Professional exam. Although they were scenario based, they were not as long and difficult to digest as the questions I remembered from my Professional exam.

Below are the topics that I encountered during the exam and video resources that can help with preparing for the exam:

AWS Cloud Watch, AWS Config and AWS Cloud Trial – understanding the differences between the services, how they all fit together into various scenarios and how they can be used with other services to automate security operations.

]]>https://pajdzik.com/2018/02/11/my-impressions-after-taking-the-aws-certified-security-speciality-beta-exam/feed/9How to get ahead as a Solutions Architect – insights for an article written for Gazprom Marketing & Tradinghttps://pajdzik.com/2018/01/19/how-to-get-ahead-as-a-solutions-architect/
https://pajdzik.com/2018/01/19/how-to-get-ahead-as-a-solutions-architect/#respondFri, 19 Jan 2018 00:27:43 +0000http://pajdzik.com/?p=607I have been recently asked to contribute my thoughts and insights for an article for Gazprom Marketing & Trading. The article offers tips and advice for those IT professionals who are looking to take the step into a solutions architect role. The full article can be read on GM&T blog.
]]>https://pajdzik.com/2018/01/19/how-to-get-ahead-as-a-solutions-architect/feed/0Why would you ever choose Microsoft Azure over Amazon Web Services?https://pajdzik.com/2017/08/16/why-would-you-ever-choose-microsoft-azure-over-amazon-web-services/
https://pajdzik.com/2017/08/16/why-would-you-ever-choose-microsoft-azure-over-amazon-web-services/#respondWed, 16 Aug 2017 16:00:50 +0000http://pajdzik.com/?p=557I have been recently asked on several occasions why I would ever choose Microsoft Azure over Amazon Web Services. There are a number of reasons why I would advise on using Azure in certain scenarios despite AWS being an undisputed market leader in cloud computing. However, the main reason is usually business alignment.

You may already use Microsoft Technologies in your business.

If you already run a number of applications on premises or in your datacentre that are based on Microsoft technologies, you probably already have an in-house expertise in these technologies. If this is so, it will be easier for your team to work with Microsoft Azure as it comes with a first-class support for Microsoft technologies such as .NET and MSSQL, simplified deployment and integration with Visual Studio. If the application you are building uses these technologies, this is a strong indicator that you may be better off with Azure as this way you would avoid skillset and recruiting challenges.

You may discover that Azure offers the best technological fit.

Even if you currently do not have much expose to Microsoft technologies, you may perform an analysis of the problem that you are trying to solve and conclude that Azure offers a better technological fit for your solution. This is an unbiased and valid reason for choosing a specific provider or technology over another. It may also be that Azure offers compliance with a certain standard that AWS does not. The fact that AWS is the market leader may not matter that much.

You may benefit from working with multiple cloud providers.

Sometimes, even if you have an in-house expertise in AWS and may already have some applications running on AWS, it may be more beneficial for you to run your new application on Azure. Many larger organisations do this for a mixture of reasons. Firstly, it may be beneficial for your organisation to be involved with multiple cloud providers as you put yourself in a better negotiating position with them when it comes to discounted pricing or support. Secondly, you may want to avoid a total dependency on one public cloud provider and prefer distributing your key applications across multiple vendors.

Although AWS is the leader in the cloud computing space, it faces a fierce competition from Microsoft. The latter has the competitive advantage of its brand and client support which has been built up over the decades in the industry. It may be just a matter of time before the gap closes.

]]>https://pajdzik.com/2017/08/16/why-would-you-ever-choose-microsoft-azure-over-amazon-web-services/feed/0Passing the AWS Certified Solutions Architect – Professional examination.https://pajdzik.com/2017/04/05/passing-the-aws-certified-solution-architect-professional-examination/
https://pajdzik.com/2017/04/05/passing-the-aws-certified-solution-architect-professional-examination/#respondWed, 05 Apr 2017 19:32:39 +0000http://pajdzik.com/?p=502I have worked with Amazon Web Services since 2008. For the past couple of years, I have done a lot of work around network design and hybrid cloud in enterprise level environments. Over all the years, I have been exposed to most of the AWS services.

The examination is not easy but someone with a few years of experience should not require a lot of time for preparation. It took me two weekends to prepare and feel quite confident. Here are a few remarks about the examination:

As for the time of writing (April 2016), the exam is not very up to date. Newer services from about the past two years (lambda, aurora, application load balancer, etc.) do not seem to be covered at all. That may change soon.

The questions are lengthy and hence they require good reading comprehension skills and ability to keep focused. In each question you are presented with a business scenario and you need to pick the right answer(s) to meet specific business requirements.

The answers are formed in such a way that more than one answer can be technically correct. Very often you are required to pick only the most correct answer but sometimes you are asked to choose two or three.

You need to answer about 80 questions in 170 minutes. You have just over 2 minutes for each question. This is not a lot of time.

After the first 20 questions, I felt confident that I would pass the test. I did not find the questions very difficult but it took time to read and understand them. After 40 questions, I started getting a bit concerned as I noticed that it took me too long to get to this point. After 60 questions, I made up a bit of time but I felt like I had to rush through the questions. On top of that, it became more and more difficult to focus. When I had gone through all the questions, I only had just about two minutes left. No much time to review so I decided to finish the exam. That was my first attempt and I passed.

I have not done any of the online courses from acloud guru, cloud academy nor linux academy. I have heard good things about them though so if you have enough time to go through them, I am sure you would benefit from them.

In order to run .NET web applications on Amazon Web Services EC2 instances in an Auto Scaling Group behind an Elastic Load Balancer, a process needs to be set up that will deploy your application each time a new EC2 instance goes online. The Octopus Deploy server is a neat tool that can be used for deploying applications to multiple environments. With a bit of setup, you can make it work very reliably with AWS EC2 instances.

Your web application will be running in a VPC. Assuming that you have (and you should have) multiple VPCs for your multiple environments, you should ideally place your Octopus server in a separate “hub” VPC as per the illustration below. In the future, you will be able to easily add new environments or applications to this setup. Your Octopus server should be set up in a private subnet that ideally is only accessible from trusted locations.

To allow communication between your Octopus server and application servers you should use VPC Peering. You also need to add a route to your route tables associated with the subnets where your application servers run. The destination of that route should be the CIDR of the subnet where your Octopus Server is and the target should be your peering connection. Similarly, you should add routes to the route table associated with your Octopus Server subnet. Your application subnets CIRDs would be the destination for that routes and your peering connection would be the target.

EC2 instances will have to be registered with Octopus server when they go online and deregistered when they are terminated. The full process is presented in the illustration below:

When a new instance is added to the load balancer, an Octopus tentacle needs to be installed on the instance. Once the tentacle is on the instance it has to be registered with the Octopus server. It will be given its Octopus machine name. We need to tag the EC2 instance with the Octopus machine name so that when the instance is terminated, we know what tentacle should be deregistered from the Octopus server. Finally we need to deploy the latest version of the web application to the new instance. All this can be accomplished by a user data script (written in power shell) passed to the EC2 instances. This script will be executed when a new instance is launched.

When an EC2 instance is terminated we need to read the corresponding Octopus machine name and make a request to the Octopus server to deregister the tentacle. This can be accomplished by a lambda function triggered by Cloud Watch event.

]]>https://pajdzik.com/2016/09/15/high-level-overview-of-automated-deployment-with-octopus-deploy-to-aws-windows-server-ec2-instances-running-behind-elastic-load-balancer/feed/4Cloud Architecture: 14 best practices for designing a Virtual Private Cloud (VPC) on Amazon Web Services – updated on 21.12.2017https://pajdzik.com/2016/08/01/cloud-architecture-best-practices-for-designing-a-virtual-private-cloud-vpn-on-aws/
https://pajdzik.com/2016/08/01/cloud-architecture-best-practices-for-designing-a-virtual-private-cloud-vpn-on-aws/#commentsMon, 01 Aug 2016 10:12:49 +0000http://pajdzik.com/?p=401As much as AWS VPC offers security and flexibility, there are certain aspects of its design that are worth proper consideration prior to implementation. This is because if you do not lay out your VPC well enough, you may not leave yourself enough room for manoeuvre when your business grows or requirements change. If that happens, you will have to spend time and money on migrating to a more suitable VPC. Below is a set of best practices I find useful when designing a VPC.

You cannot change the IP CIDR block of your VPC after creation. If you realise down the line that your VPC is too small you will have to create a new one and migrate your instances. Make sure you plan for business growth and think a few years ahead. Currently, sizes between /28 and /16 are supported. If unsure, create a larger VPC rather than smaller.Updated on 21.12.2017: You can now add up to 4 more CIDRs if you need to expand your VPC. It is not possible to change the original CIDR though. Also, the original CIDR dictates what other IP spaces can be used. For example, if you use 10.x.x.x/24 than you can only add additional spaces from the RFC1918 10. space. To summarize, there are still some restrictions but if your outgrow your VPC, you can now just add another CIDR rather than migrate to a new VPC or work with multiple VPCs

Make sure that your VPC IP CIDR does not overlap any network addresses in your office or your data centre. Also if you use multiple VPCs for application and / or environment isolation (recommended), make sure that you do not use the same or overlapping IP CIDRs between them. This is to avoid issues when you later want to connect your VPC via VPN to your company networks or if you want to use VPC peering to connect multiple VPCs.

The minimum size of a subnet is a /28 (14 IP addresses). Remember that AWS reserves the first four IP addresses and the last 1 IP address for internal networking purposes. Note that some AWS services may require a few IP addresses. For example, elastic load balancers require a number of IP addresses to work and the number will change over time to support scaling and failover. Make sure you give yourself enough space.

Sizes of your subnets should be decided based on their purpose. If you come from a traditional networking background, remember that traditional subnet constraints such us broadcast domain limits do not apply in a VPC. Multicast and broadcast are not actually supported. You can create subnets with 1000s of instances if you need to. Again, make sure you plan for business growth and think a few years ahead.

When you create a subnet, it will be associated with the main route table automatically. If you need different routing for that subnet, create a new routing table, associate it with the subnet and modify the new route table. Never modify the main route table otherwise other subnets may be given routes they should not have upon creation.

There is a default local route in every VPC routing table that enables intra subnet routing. The route cannot be deleted and you cannot create a more specific route than this. That means that the network within your VPC is flat regardless of how many subnets you have created and all resources can talk to each other. You separate resources using security groups (stateful firewalls) and NACLs (stateless firewalls). In terms of routing you only define ways out of the VPC (Internet Gateway, Virtual Private Gateway, NAT, VPC Peering, VPC endpoint) and not routes between your subnets.

A subnet in AWS is not an isolation container for an application. Subnets should be instead decided based on their routing needs. For example, public subnets with a route to the Internet Gateway, VPN only subnets with a route to Virtual Private Gateway, Private subnets with a route to NAT gateways only, etc… Apart from routing considerations, you should, of course, create subnets across multiple availability zones for high availability (see point 4). You may also have specific security requirements that will require a network ACL on a specific address space. That could also constitute creating a separate subnet.

Use Internet Gateway to enable internet access in your public subnets. Do not put application servers in the public subnets if they are to run behind Elastic Load Balancers unless you have a very good reason (for example bandwidth intensive processes). In most cases you should put ELBs in public subnets and application servers in private subnets for enhanced security.

Network ACLs operate on the subnet level and do not filter traffic between instances in the same subnet. They are stateless which means that you will have to implement two rules. This adds extra complexity and is difficult to maintain. Use network ACLs sparingly to enforce baseline security policy by blacklisting.

Security groups operate on the instance level. They are stateful and easy to configure and if you keep them simple, they are easy to maintain. They can reference other security groups in the same VPC. They should be your main tool to secure your VPC.

Use VPC Peering so that multiple applications in separate VPCs can communicate securely. Traffic between instances in peered VPCs is private and isolated and there is no single point of failure or bandwidth bottleneck. VPC Peering can connect VPCs in multiple accounts but only within the same region. If you have applications running in multiple AWS regions, you will need to use VPN for secure communication.Updated on 21.12.2017: AWS has now announced VPC inter-region peering that encrypts with no single point of failure and no bandwidth bottleneck. Traffic is always routed through the AWS backbone network hence provides a high degree of isolation.

Use NAT gateways if you need instances in a private subnet to communicate with the internet. For example if they need to access public AWS services such as dynamo DB or third party endpoints. NAT Gateways are fully managed, redundant, highly available and can handle up to 10Gbps of bursty traffic. The downside is that they do not offer (unlike Internet Gateways) multi-zone capability (A NAT Gateway has to be created per availability zone) and require elastic IPs.

Use VPC endpoints to access S3 resources from within your VPC. They allow you to access to Amazon S3 resources (AWS promises to add more services at some point) securely from your instances without having to use Internet Gateway, VPN, or NAT. Both a VPC endpoint and an S3 bucket have to be in the same region though. Use IAM policies to control VPC access to Amazon S3 resources.Updated on 21.12.2017: AWS has released a new service called PrivateLink which is a collection of are API endpoints for a number of AWS services that can be added to your VPC and accessed securely as private endpoints.

A VPC (Virtual Private Cloud) is a Software Defined Network (SDN). It is an AWS service that provides AWS customers with means to provision a logically separated section of AWS Cloud. By creating a VPC you are creating your own network within AWS.

You may have to design multiple VPCs for your projects and environments. Your VPCs are the fundamentals of your cloud infrastructure. If you create a solid, future-proof design now, you will avoid expensive and time-consuming adjustments that may disturb your business operations when your business grows. How you design your VPC will have a direct impact on flexibility, scalability, security and fault-tolerance of your cloud infrastructure.

In this article I will briefly present VPC design considerations for an n-tier web application:

Choose your Virtual Private Cloud size and network CIDR wisely

The first thing to consider is the size of your VPC. The allowed CIDR block size is between /28 netmask (16 IP addresses) and /16 netmask (65536 IP addresses). It is important to realise that you cannot change the size of your VPC once you have created it. If you run out of IP addresses, you will have to delete your VPC and create a larger one. That, of course, means that any EC2 instances and other resources will have to be terminated and re-launched. Therefore, to achieve as much flexibility as possible, I allocate the maximum possible number of addresses to the VPC.

For the sake of this example I will use 172.20.0.0/16 network that gives me 65536 addresses and an address range of 172.20.0.0 – 172.20.255.255. It should be noted that any potential conflicts with other networks that may be connected via virtual private gateway to your VPC should be considered. CIDR should be chosen to avoid such conflicts. You do not want to use 10.0.0.0/16 network if your local datacentre uses 10.0.26.0/24 for example.

/16 network with its 65536 addresses may appear too big for you especially if you have multiple networks within your organisation occupying various private IP address spaces. If /16 seems to you like a waste of precious space and you think that you will never need the enormous number of IP addresses in your VPC, consider smaller networks such us /18 or /20.

Geographical locations matters

I assume that the majority of my visitors will be coming from the UK. I will therefore build my infrastructure in the EU west 1 region (Ireland) due to its closest proximity to the UK. There are currently three availability zones (different physical locations) in the EU west 1 region. To achieve the highest possible fault tolerance in this region, I will use all three availability zones. I divide the address space evenly across them but also leave a spare block of IP addresses to keep the set up flexible:

172.20.0.0/18 – Availability Zone a (172.20.0.0 – 172.20.63.255)

172.20.64.0/18 — Availability Zone b (172.20.64.0 – 172.20.127.255)

172.20.128.0/18 — Availability Zone c (172.20.128.0 – 172.20.191.255)

172.20.192.0/18 — Spare (172.20.192.0 – 172.20.255.255)

Each availability zone has now 16382 IP addresses and there are 16382 spare IP addresses. I keep the spare addresses just in case. I may allocate them in the future should it be required.

Subnets layout is important

I place AWS resources in subnets depending on how they need to be accessed. Generally speaking, it is a good idea to create a subnet if there is a need for different hosts to route differently or if it is advantageous to use network ACL on a subnet level. Subnets cannot span across availability zones so I will have to create a subnet in each of the three availability zones per each tier of my application. Taking all that into consideration I create three subnets in each availability zone:

Public subnet – only public facing resources that requires full internet access should be deployed into this subnet. In the case of my application only elastic load balancers will be placed in this subnet.

Private subnet for application tier– EC2 instances hosting the web application will be placed in this subnet.

Private subnet for data tier – any data stores such as RDS and ElasticCache will be placed in this subnet.

I will have 9 subnets in my VPC (3 tiers x 3 availability zones) I will allocate the same number of IP addresses with each subnet and I will still leave some spare IP addresses in each availability zone to keep things flexible. I will therefore use CIDR /20 subnets which will give me 4094 IP addresses in each subnet. Therefore, there will be 12282 IP addresses per tier (3 availability zones x 4095 IP addresses each).

You may of course, get to a different conclusion when you consider your n-tier web application. For example:

Larger App tier

You may conclude that it is more likely that your application tier will contain much more resources (EC2 instances) than any other tiers. In this case you may prefer having larger app tier subnets and smaller web tier and data tier subnets. You could therefore lie out your subnets the following way:

Of course you would replicate the same structure across all availability zones as in the previous design.

One set of subnets for application and data tiers

You may conclude that regardless of how many application tiers you have, your VPC only needs one public tier (accessible from the internet) and one private tier (not accessible from the internet). You will launch your load balancers into your public set of subnets. Your application EC2 instances and databases will be in the same private set of subnets. There will not be a separate set of subnets for the application and the data.

This kind off flat private tier that contains mixed resources is absolutely fine if you do not have any particular reason to split your resources into separate subnets. It also has one big advantage. Namely, you won’t waste IP addresses that otherwise would have to be assigned to subnets but not actually used. Think about auto-scaling groups. You may have many EC2 instances but fewer RDS instances however you will never know the exact numbers. If you create two private tiers, you will simply have to guess the numbers like in the previous example where we created our application tier to be twice as large as our data tier. If you create one private tier, all the IP addresses can be allocated to any type of resource.

Again, you would replicate the same structure across all availability zones.

Routing considerations

All instances in the VPC can communicate with each other by default. We do however have public and private subnets that require different routing tables.

Resources in the public subnet needs to be directly accessible from the internet. I therefore create an Internet Gateway in my VPC and a route table in which the Internet Gateway is the default route. I then associate the route table with all three public subnets. I only launch elastic load balancers into the subnets as these are the only resources, at least initially, that should be accessible from the internet. There is no need for any application EC2 instances to be in the public subnets (unless there is a good reason, for example bandwidth intensive processes that struggle behind NAT instances or gateways). They actually should not be there as normally they are only meant to accept traffic from load balancers only.

Although instances in the private subnets should not be directly reachable from the internet, they require outbound internet access. Normally, I would create NAT instances in each of the public subnets however by the end of the last year (December 2015) AWS released managed NAT gateways. Using a managed NAT gateway is simpler than creating your own NAT instances, therefore in most cases I would now go with this option. Unfortunately, unlike the Internet Gateway, NAT gateways do not have the same multi-zone capability. Therefore, I have to create a NAT gateway in each of the public subnets and create route tables with routes to the NAT gateways that I associate with private subnets.

There are other routing options that you may need to consider when designing a VPC for your n-tier web application. You may need a VPN connection between your office or your data centre and your VPC. In this case you have to create a customer gateway and virtual private gateway. Then, you create a route to your virtual private gateway in the subnets where you will launch instances that should communicate directly with your other networks.

You may also want to create and route traffic to VCP endpoints. They allow you to direct traffic from your VPC to S3 (and other AWS services in the future) directly without having to access the service over the internet.

In more advanced set-ups you may want to use VPC Peering (allowing communication between two VPCs) in which case you will also have to configure your routing tables to enable communication.

All these routing options should be kept in mind when you first design your VPCs. If you properly allocate IP addresses in the first place so that they do not overlap any other networks or other VPCs and so that you have enough room for growth in each of them, your business will have a strong fundamentals in the cloud.

]]>https://pajdzik.com/2016/04/12/network-vpc-design-considerations-for-a-flexible-secure-scalable-cost-effective-and-fault-tolerant-n-tier-web-application-on-amazon-web-services/feed/10Behaviour Driven Development is a communication framework. Don’t use it just for testing!https://pajdzik.com/2015/12/16/behaviour-driven-development-is-a-communication-framework/
https://pajdzik.com/2015/12/16/behaviour-driven-development-is-a-communication-framework/#respondWed, 16 Dec 2015 20:41:50 +0000http://m.pajdzik.com/?p=275Many companies use Behaviour Driven Development as a part of their software development process for automated testing of their business logic. The idea of Behaviour Driven Development comes from Test Driven Development. In TDD, developers would normally write tests in a programming language. The tests would cover some areas of the application functionality. Because of that, these tests would only be understandable to the developers (or perhaps also to some technically minded QA engineers). However, non technical stakeholders, would not be able to read these tests.

BDD, unlike TDD, requires you to write test scenarios that describe the desired system behaviour in a human readable language. Such tests, however, cannot be executed and hence a second, corresponding set of tests needs to be written in a programming language. The second set of tests is called step definitions and these are the tests that get executed. Clearly, to practice BDD you need to write your the same tests twice!

This increases the time required for writing the tests and increases the cost of maintenance. Therefore, if you are only using BDD for automated testing and if your developers are the only people who look at the tests, then you are wasting your time and money. Perhaps a methodology that would not require you to write your tests in two versions, would be more appropriate and certainly cheaper. There is however a good reason for investing in BDD, provided you use it in a way that allows you to take advantage of all the following benefits:

Enhanced communication and collaboration

It should be well understood that Behaviour Driven Development is first and foremost a communication framework. The main point in practising BDD should be to allow not technical people who hold a stake in a software development project (project managers, business analysts, non-technical QAs, etc.) access and understand the tests. The tests are written in a human readable language and therefore they can be used to facilitate communication between technical and non-technical members of the team. BDD scenarios help establish a single version of truth.

Better visibility and ability to prioritise.

Like in Test Driven Development, BDD tests are supposed to drive the development. However, because they are written in a human readable language, they provide non-technical team stakeholders with better visibility of the software development process. It is easier for business stakeholders to assign business value to each of the tests scenarios and prioritise development this way.

Documentation

Since BDD tests are written in a human readable language and since they define behaviours of the system, they can be used by both technical and non-technical stakeholders as a documentation of the software projects.

Quality assurance

Last, but certainly not least, BDD scenarios can be used for testing the business logic.

As you can see, testing, as much as it is important, is only one of the benefits that BDD offers. The main idea behind BDD is to facilitate communication between all stakeholders.

]]>https://pajdzik.com/2015/12/16/behaviour-driven-development-is-a-communication-framework/feed/0Business perspective: Is project based outsourcing to India economically viable?https://pajdzik.com/2012/11/21/business-perspective-is-outsourcing-to-india-worth-it/
https://pajdzik.com/2012/11/21/business-perspective-is-outsourcing-to-india-worth-it/#respondWed, 21 Nov 2012 12:07:35 +0000http://m.pajdzik.com/?p=328The outsourcing landscape is changing. US and European economies are still fragile. The pressure is on for IT departments to further cut costs, whilst maintaining existing systems and supporting revenue growth through technological innovations within a global marketplace. Yet larger multinational companies are starting to move away from outsourcing of IT operations and developments and returning to more traditional in-house development models, with home grown resource. Coupled with the news that Indian outsourcers reporting a downturn in revenue and profits figures due to adverse economic conditions in the US and Europe[1] provides a golden opportunity for the agile CIO. Small or medium sized corporations are strongly positioned to leverage the benefits of project based sourcing opportunities, for developments in niche technologies, in a falling outsourced market.

The recent news[2] that GM CIO Randy Mott intends to bring back 90% of his IT organisation in-house over the next three years with an potential cost of over $1billion is seen by many industry observers as a radical and risky approach to the daily challenge that faces every IT Manager and CIO in delivering cost effective, time sensitive and quality driven technologies to achieve real business benefits. Mott believes that the company needs ‘more creative, business-changing ideas from IT, and IT teams need to deliver those innovative projects faster’. Mott believes that GM cannot be creative or fast enough with outsourced IT. To a certain degree Mott is right, intellectual property, business knowledge and strong project management skills are critical skill sets that need to be maintained in-house to protect the integrity of a company’s business and technical infrastructure. But where does that leave the smaller IT Department who doesn’t have unlimited funding, maintains a small internal headcount, is working flat out with limited resource to maintain existing legacy technologies, but still needs to support fast moving technical deployments, to ensure commercial advantage in the digital marketplace? Tactical outsourcing of specific niche skill sets to India, assuming the specifications reflect the business need, that the development is controlled within a workable methodology, ant there is open communications, still fits within a pragmatic and cost effective outsourcing strategy.

Traditionally, large corporations are resilient to change and the risks of the smallest deviation for imbedded processes and procedures, unless managed tightly, are well known. But for the small or medium sized company proactive change is the lifeblood of success. Adopting new technical innovations, using an agile outsourced framework, abet in a risk-controlled environment, and could provide that much needed low cost competitive edge. The ability to respond quickly to new market trends, quickly place product promotions, support open online communications combined with successfully entry and exploitation of new and diverse digital channels is something every CIO and IT manager should aspire to. Even maintaining equilibrium with competitors in the digital arena required migration to newer enabling technologies as some point in time otherwise maintenance costs become disproportional to investment returns. CIO magazine tells us that for over 50% of IT leaders their major challenge today is increasing speed of developments and keeping down costs.

Yet in many cases, the development and maintenance of new innovative technologies, through the creation of an in-house team is not a financially viable business model for the smaller company. With the current scarcity of skilled permanent resource costing upwards of £35,000 per annum, the overheads can soon mount up for a fairly complex deployment. As larger companies are moving away from sourcing through India and looking to recruit new graduates as permanent employers, even recruiting inexperienced resource will cease to be an option for the smaller company. Whilst building up the expertise of in-house staff may be a delivery solution in the longer term, the development of resource and training overheads, coupled with the risks of using inexperienced staff in a small fast moving environment, should not be underestimated. For example a basic two-day introductory training course for a Drupal developer is close to £2000 alone. Coupled with a need for enhanced supervision and guidance of trainees, the risks of successfully building an in-house team from scratch, utilising leading edge or niche technologies, are obvious. So what other delivery options are available?

The cost of developing a professionally created web management system can soon mount up, without taking into account the ongoing maintenance charges on a monthly basis. Resourcing estimates show that a successful, fairly complex Drupal CMS based project would realistically require between 3-4 developers. Nigel Shadbolt[3] provides us with an insight into the development timescales and resource estimates required to create the new pan-European government data proposition, highlighting the benefits of using open source architecture. As a prominent adapter of this framework, data.gov.uk[4] suggests a team of 3 fulltime coders and a designer were required for their initial iteration, subsequently retaining 2 internal Drupal developers for module updates, customization and monitoring of the application. In other cases, a figure of 30 man-hours per month is suggested as a minimum estimate for ongoing maintenance.

The minimum rate for a good Drupal contractor in the UK (working on site) is around £350 a day, with city based consultants achieving between £500-600 and more. For the smaller IT department the business case, to adopt a traditional in-house development framework or out source locally, just does not stack up. With high costs of training for development staff in new technologies and the costs of bringing in expensive contractors on site (if they are available) could leave the business compromised by the IT department’s inability to respond to pressing business needs in the digital arena. So how can an alternative sourcing strategy, which moves specific parts of a delivery to remote developers based in India, work in practice?

As a recent working example, we could consider the design and implementation of a mid-sized content management system in the private sector. A UK publishing company, with limited in-house technical expertise, required a full development of a CMS within a six month period, to meet a major business launch date. Having secured an experienced consultant, with web implementation skills, to help define the business requirements and supporting technical architecture, a variety of delivery options were considered. Realistic estimates indicated the project needed three developers for a 6 month delivery. Consideration was firstly given to hiring two contractors and one permanent employee, who would then stay and maintain the project. With no applicants available with enough Drupal expertise and the least expensive contractor charging £350 a day, the development costs alone were over £136k for the 26 week period. By utilising contacts with an Indian development company, three remote developers were offered, interviewed and accepted at a cost of 30k. The project was a success. After the development the company retained one of the Indian developers to support the application for another 9 months, whilst a UK based fulltime developer was found and trained to the standard required for ongoing maintenance.
Critical to the success of this project were the following criteria:

Open Communications: An understanding of how to manage and communicate with the developers in India was important. Although the common language was English it was easier from the publishing company to adjust their communication style to align with the Indian development team. Key disciplines included ensuring that there was clarity of instruction and common terms were agreed up front. This was the steepest learning curve for the UK Company and therefore they retained the services of the Business Consultant who had previous experience of outsourcing with Indian developers. Skype conferences proved invaluable to maintain constant contact and ongoing management controls.

Management style: Inclusive. The UK management team welcomed input and comments from the developers on an ongoing basis. Feedback on the UK technical design proved invaluable in establishing a common understanding of the requirements.

Agile development: A modular approach to delivery was adopted that allowed for small, version controlled releases.

Service Management: Strong service management disciplines were required supported by a ticketing system, version control and testing software an ITIL approach was taken. A performance management regime was put in place. Feedback on each deliverable was provided to the developers, to foster an environment of continuous improvement.

Elements of the project retained in the UK: Business Analysis, Solution Architecture, Project Management and Quality Assurance roles.

CIO BOTTOM LINE: Tactical project based outsourcing to India should be considered for mid-sized web developments. Opportunities of up to 80% financial savings compared to equivalent UK based developments have been achieved. Recognition that there needs to be strong project management skills in place to control outsourced deliveries is paramount. Mitigation of unsuccessful delivery can be achieved by retaining analysis, design and management disciplines within the client company.