Archive for February 2012

I’ve blogged a few times about using SData to feed data into Sage products and about the general ideas behind SData. In this blog posting I want to look at some benefits to integrating into Sage products using SData and the general reasons behind doing so. A lot of this article is forward looking and so each Sage product is somewhere along this journey and no product has reached the end yet.

Imagine that you are an Independent Software Vendor (ISV) and that you have a great product that satisfies a useful specialized business need for many customers. However it requires some data that is typically stored in an ERP system and generates some financial results that need to be entered into this ERP. Say you need to read all the Orders from the system and then you produce a number of G/L entries that need to be imported. Wouldn’t it be great if you could automate this process by being notified automatically whenever an Order is entered and being able to write the G/L entries directly into the ERP? However, you find that your customers are running all sorts of different ERP systems. As you look into doing these integrations you find that you need to join an SDK program for each ERP and have someone learn and program in all these different ERP’s SDKs and program you integration in all sorts of strange languages from VB to C# to JavaScript to FoxPro. Then you have to maintain all these integrations from version to version as your product progresses and all these ERPs evolve. This is a lot of work. There must be an easier way.

You could take the attitude that you will provide an API and SDK for your product and then, since your product is so wonderful, that all the ERP vendors will write an integration to your product using your SDK. After a few years of waiting for this to happen, you’ll probably give up on this.

Sage has many ERP systems and combined has a very large market share. However all these ERPs have their own heritage and their own SDK. How can Sage make it so this ISV can create their integration easily to a large set of Sage products using their own development tools and programming knowledge?

This is where SData comes in. SData is a REST based Web Service interface. SData is based on internet standards and easy to call from any programming system that supports Web Service calls. SData is a standard being implemented in all Sage Business Solutions. So how will this work in the scenario above?

Method One – Poll for Orders

In this case the ISV application will periodically poll for new Orders on the Orders SData feed, process them and then write any G/L entries to the G/L SData feed.

This method isn’t that different than using the ERP’s built in API in whatever technology that is implemented, perhaps COM, .Net or Java. So at least now for the various ERPs we are using only one API technology to access several ERPs.

Standard Contracts

Generally each ERP has its own heritage and hence its own database schema. Most Sage ERP’s will expose their built in native schema as their native SData schema. So although you can use SData to access a range of different ERPs, you still need to handle multiple schemas to set and return in the XML payloads associated with the calls.

The solution for this is standard contracts. These are specifications for standard SData calls that define a common schema. It is then the job of each Sage ERP to transform their native schema into the schema of the standard contract as part of their job of offering standard contract feeds.

With standard contracts implemented the ISV can then access multiple Sage ERP’s using the same API technology, namely SData, and using the same payload schemas, essentially making the integrations the same.

Method Two – Connected Services

Connected services refer to integrating products whether from Sage or an ISV being located in the cloud and then integrated to an on-premise installed ERP.

Currently there is a concern that on-premise customers don’t want to expose their ERP system to the Internet, meaning they don’t want to manage a firewall and worry about the security concerns associated with an Internet facing web server. This means they do not want to expose their ERP’s SData feeds to the Internet at large. So how does the ISV integrate if they can’t call the ERP’s SData feeds?

Although the cloud application can’t call the on-premise application, the on-premise application has no problem calling the cloud application. This means it’s up to the on-premise application to push all necessary data to the cloud application. Generally this is handled by a synchronization engine that synchronizes some ERP data up to the cloud. The SData specification contains a data synchronization specification for this purpose. The synchronization algorithm is very robust and will catch up if the Internet is down, or the cloud service is unavailable.

We could also provide notifications when things happen, but this is less robust, since what happens when the Internet is down or the other application is unavailable (perhaps offline for maintenance).

Notification Configuration

The above method for connected services assumes that SData synchronization will need to be configured. The core ERP will need to configure a list of SData services to synchronize with (hopefully down the road there will be many), then the ERP will need to ask the service which feeds it wants to synchronize.

Initially this configuration will probably be done by configuration screens in the ERP. However over time it will be nice to move this all to dynamic discovery type services. This will reduce error in setup and allow connected services to evolve without requiring setup changes in all their integrated ERP systems each time something new is added.

If the ERP and ISV solution are both installed together (either on-premise or in the cloud), then it would be nice to configure some direct notifications, basically call-outs from the application when various important things need to be done. This might be for things like requiring a calculation be performed or to notify that an order’s status has changed. These sorts of things will only work when both applications are live and connected, but there are many useful scenarios for this.

Summary

Initial SData implementations are either shipping or forthcoming across all the main Sage business products. We are starting to see integrations of the method one type emerging. We are actively developing connected services using the SData synchronization mechanisms. Then as we continue to roll out SData technology you should see any missing pieces start to fill in and greater uniformity emerge on how you integrate with all Sage products.

As Sage studies the market and determines useful profitable connected services, Sage can publish an SData contract that all our ERP systems will support. Then the chosen ISV can provide this service for all Sage products by supporting this SData contract. Often for connected services we need different providers in different parts of the world, and again it’s easy for Sage since for this one contract perhaps we have one provider for North America, another for Africa, another for Asia and another for Australia.

This article contains a number of pitfalls that I’ve run into as part of taking a large established development team from the Waterfall to the Agile Software Development Process. This involves taking a development organization with around 100 professionals in various roles including programming, quality assurance, business analysts, user centered design, product management, documentation, project managers, people managers, etc.

Through it all its best to always refer back to the Agile Manifesto as a guide to whether changes are helping or hurting:

Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan

Some of the items discussed here are controversial and I’ve certainly have colleagues who think these are good things. So keep in mind these are my personal experiences and judgments.

Too Much Process

Generally the solution to most problems with Waterfall involves adding more process, such as additional checkpoints and oversight. There is a huge tendency to carry this sort of thinking to Agile. It is totally alien that something can work without adding back all these layers and processes. Empowering teams to produce working software rather than documents and plans is always a challenge.

Checklists

As people are new to Agile, they go through various training classes and read various books on the Agile process. To help with the learning they develop checklists, such as things to do at standup or things that need to be done to mark a story as done.

Creating a checklist of all the things that need to be done to complete a story seems like a good idea and usually has useful items like performing a code review or making sure unit tests are in place. However then blindly applying these checklists to all stories causes all sorts of unnecessary work. It also tends to set people up as gatekeeper which tends to be a bad thing. For instance a story that involves configuring some XML configuration files for a server component shouldn’t need any user centered design review, or Java code review. However when these are on the checklist, people blindly assume they need someone with these expertise to sign-off to complete the story. Or they need a special dispensation to skip the item. All of this leads to unnecessary discussion, meetings and wasted time. Worse it leads to feature trading negotiations. Say one of these reviewers doesn’t have anything relevant for this story, but want something low priority done somewhere else, and then they will “trade” their approval for this other work being done. This then subverts the product owner’s prioritization of the backlog.

In general these are symptoms of the product owners and scrum masters not having enough authority, trust or power. The scrum master must be diligently and aggressively removing check list items that aren’t appropriate for a situation. Management has to trust the scrum process and team empowerment to believe that the team will do the right thing and apply necessary done criteria, but leave off everything else.

Over-Swarming

Most agile books advocate creating a priority ordered backlog and then having teams swarm on the top items to get these most important items done first and done quickly. This is all great, but when you have a large development organization say with twelve scrum teams, then having this many people swarm on one problem leads to everyone tripping over everyone else.

I think you need to keep the whole swarming thing within one team. If you want multiple teams swarming on something, then there is something wrong with your backlog and it needs to somehow be refactored. Much more important than swarming is keeping things in the backlog independent of each other and really minimizing dependencies.

I think otherwise you get people thinking along the lines that if it takes a woman nine months to have a baby and Product Management has promised that baby in one month then you need to swarm on the problem and have nine mothers produce the baby in one month. Generally be careful that you aren’t really just slowing things down and distracting the people doing the real work by throwing resources at a problem.

Not Decomposing Problems Enough

There is a huge tendency for an organization that once did waterfall, when switching to agile, to keep doing waterfall, just fitting the tasks into sprints. A clear telltale for this is when they have a coding story that takes one sprint and then a QA story for the next sprint. This is really an attempt to separate programmers from QA and not have them working together on a story. The programmer is just throwing the code over the wall (or sprint end) to the QA just like the good old waterfall days.

The usual excuse for this is that the story is intrinsically to complicated or intertwined to break down and must be done over multiple sprints. Don’t believe this. Every case I’ve looked into, that did this, could be quite easily decomposed into multiple stories. It’s just a matter of adjusting your thinking to work that way.

I don’t like forcing rules on scrum teams, since this undermines their authority to do what is best for a story, but this is the one case where I find this necessary. If this isn’t done then the stories just get larger and larger until waterfall is back to being the norm.

I really emphasize releasable state, that these big stories are taking you out of releasable state and that this is a very bad things.

Source Control

Another problem I’ve seen is that during sprints, teams are checking things madly into the main source tree so that the continuous build and integration system picks them up and their QA can immediately test what they have done. However this can be disruptive to other teams, if one team is breaking things mid-sprint.

If this starts happening, more use of branch/merge needs to be done. With Git this is very natural. With Subversion, this isn’t too hard either. In any case each team needs to cognizant of not disrupting other teams as they race to complete their own stories. Modern tools have all the features to facilitate this; you just need to educate people to use them. Basically if your changes are breaking other teams, then work in a branch and don’t merge until your testing is complete. With test driven development make sure the testing is in place and working before committing to the trunk.

Summary

These are some of the problems that I’ve run into with Agile. I find that as long as teams stick to the intent of the Agile Manifesto, then they can be extremely productive and produce very high quality work. The key is to empower the teams, stick to the basic Agile principles and avoid too much process or extra bureaucracy.

I’m writing this blog posting from South Africa where I’m attending and presenting at Sage’s African Insights conference. This conference is primarily for the business partners that work with the Sage Accpac, X3 and CRM products. In Africa, Sage sells a number of small business solutions like Pastel and Act!; however, they have their own conference. Africa is a fascinating continent where you get to have many adventures you couldn’t experience anywhere else. If you like photography or wildlife then Africa is the ideal place to visit. The people are incredibly friendly and fun.

Conference

The main theme of the conference is the release of Sage X3 Standard Edition. This edition includes a methodology to standardize the implementation of X3 sites. In Africa there aren’t really any Tier 1 companies. As a result Sage has products that cover the whole spectrum of companies in Africa with Pastel covering the small business market, Accpac covering medium sized businesses and then X3 covering the larger enterprises. I’m not using the new product name for Sage ERP Accpac in this article, since these product naming changes won’t be introduced into Africa for another year.

Africa is Sage’s highest growth region showing double digit growth in all areas. The economy in many regions of Africa is booming. Peace has broken out in many southern countries including Mozambique and Angola allowing phenomenal economic growth. Africa is extremely natural resource rich and as mineral prices rise, so do exports. Hopefully this new wealth will spread widely enough to keep this positive change happening.

All the African partners are very hard working and resourceful. Generally the size of customer sites are larger in Africa than other regions, however sometimes the infrastructure to support these is a bit lacking. It’s amazing what can be accomplished though, working around all the various problems with unreliable power and internet connectivity.

This African conference is nice because there is such a sense of community. All the people present are working with the same Sage products and the conference with 300 or so attendees is still small enough that most people know each other. Even though there are many day to day rivalries between the various partners, they still come together as friends at these conferences to share their battle stories and to have a drink together. It’s also fascinating to meet partners from all over Africa, from such interesting and exotic countries.

Day 1

The conference started with a number of keynotes including Jeremy Waterman presenting the overall state of the business, myself presenting changes to the Sage ERP Accpac roadmap (a topic of future blog posts), Lorcan Malone and Tom Nolan on Sage CRM, Benoit Le Tohic and Jeremy Cocqueel for Sage X3, Keith Fenner with the sales keynote and then Mandy Wiener, the author of “Killing Kebble, An Underworld Exposed” as the guest speaker.

This then was followed by a reception in the exhibition hall where Sage and many third party solutions had booths showcasing their solutions. It’s always nice to see such a variety of third party solutions. Some have been integrating to Sage products for over twenty years.

Sage ERP Accpac is often criticized for not having a built in workflow engine. However this gap is then filled in by a number of third party solutions. On display were solutions written in the Accpac SDK, the Sage CRM SDK and written as external orchestration programs. I wonder if Accpac did have a built in workflow engine, then would all these terrific solutions exist? Would the customer have the same breadth of solutions available to them?

Day 2

Keith Fenner started day 2 with a sales keynote on how Sage is successful at winning sales over the competition and highlighted key selling points against the competition’s many weaknesses.

Then there were many breakout sessions for everyone to attend. I gave presentations of “Troubleshooting the Accpac 6.0A Portal” and “SData: An Overview”. Then I backed up Andrea McGowan in her presentation on “Customizing the Accpac 6 Portal”.

At the end of the day was the “fancy dress” party which was themed around spies and detectives.

Day 3

Keith Fenner started day 3 with another sales keynote. Then there were another set of breakout sessions.

At the end of the day was the black tie partner awards dinner. It’s good to see so many partners achieving silver, gold, platinum and diamond status. Not only Sage gave out awards, but many of the third party developers gave out awards for the partner that either sold the most or had the best installations of their product.

Summary

Africa is always an amazing place to visit. This is my fourth trip to this conference. I always have a great time and feel extremely welcome. I always learn a lot at the conference and enjoy all the various discussions. I always feel I have gained so much good information to take back to Vancouver to share with my colleagues there.

The view of Cathkin Peak from my hotel room at the Champagne Sports Resort:

I’ve talked a bit about SaaSifying Accpac and SaaSifying Sage. Today I’m going to talk a bit more about continuous deployment which is one of the end goals of these themes. This is the notion that there is an automated process that will take code from the programmer, build a product, test it and then deploy it so that customers are running it immediately. If you’ve worked in software development, this can be a scary notion, since you’re lucky if programmers check in something that compiles, let alone something you want customers running without a lot of checks and balances. The key to continuous deployment is automating many manual processes that used to be very repetitive, time consuming and error prone.

A key to successful continuous deployment is to always be challenging the status-quo. Don’t accept that something absolutely requires manual intervention. Work hard to eliminate all gate-keepers. In a way continuous deployment is a mindset and process where you are always looking to improve your automated processes, eliminate manual processes and get features to customers as soon as the programmer commits their changes. Part and parcel with this is breaking major functionality down into many small features that are all deployed as they are done. Breaking things down into very small pieces is the key to success and the key to speeding up development. Implementing things as one big piece tends to be hard, complex and leads to customer problems.

Building the Product

Generally with modern source control systems, programmers should only check things into the main trunk once they have completed something including unit testing, automated tests and manual testing. If they need intermediate check-ins then they should use a code branch and merge that in when they are finished. However with large development projects, you can still get conflicts and unexpected results. You want to find these as quickly as possible and fix them right away while things are fresh in your mind.

The programmer checking in code is the beginning of the continuous deployment process. If they have been successful in coding and testing a useful feature for your customers, then you would like this to be available to your customers in a very short order. I blogged on the desirability of doing this here.

Now you have a build server running somewhere, perhaps being controlled by a continuous integration system such as Jenkins. The system on this server will be scanning source control for any committed changes, as soon as it sees one it will get it and build it and run all unit tests. Further it has a tree of all the dependencies in the system (usually using Ivy) and will build all dependent modules and run their unit tests. If anything fails then a number of designated people will receive e-mails that the build is now failing. The person that checked in the changes will also be notified. It is now up to them to quickly fix things (the system can be configured to automatically roll back whatever they checked in, but usually the build system just keeps using the last good build).

The key point here is that you find out right away that you broke the build. This is in contrast to say doing nightly builds, where you don’t know until the next day, and there may have been many check-ins so it can be time consuming finding the real culprit. Plus it now interrupts whatever you were meant to do the next day.

Continuous Test Server

Usually beyond unit tests, you have a continuous test server that may be using Fitnesse to run Selenium UI based tests. These tests are too long to be unit tests and would slow the build server down. However you want builds to be regularly deployed to this server and the full set of UI based automated tests to be run. Perhaps after it runs the full suite it would re-deploy the latest build (if newer) and run all the tests again and just keep continuously doing this. If a test fails it would immediately notify the appropriate people and demand prompt action to resolve the problem either by fixing the problem, fixing the test or rolling something back. Again the goal here is to detect problems as quickly as possible and to force them to be resolved quickly, not allowing problems to pile up. Generally you want to keep your product in a releasable state as I blogged about here.

Automated Processes to Deploy

Next we want to start deploying for various users to start playing with the system. So we have a build that has passed the unit tests and the automated longer running tests, now we start deploying for review by various departments. Usually this will mean automatic deployment to a server that various interested internal parties know how to access, perhaps with a common URL. Perhaps the build is deployed to various manual testing computers. Anyway at this point the system can be played with by anyone that needs to review things like User Centered Design Analysts, Business Analysts, Product Managers, other interested developers, executives, etc.

This step is actually optional. If your unit tests and other automated tests are good enough and if your development organization is confident enough and has a good track record then this step can be skipped and you can deploy directly to customers. Eliminating this step is a primary goal of many continuous development teams. Sometimes it’s just a matter of keeping one last human checkpoint in place, but there are many organizations that deploy to the production servers several times a day without needing this step. This is the fastest way to immediate customer feedback on what you are doing.

Deploy to Customers

Now the fun part, deploying to the production servers for customers. You want this to be the same process as described above so that you know it will work properly. Sometimes you might deploy to one server and have it start serving a set up customers to ensure all is well before allowing the build to go to all servers. Key to this is the ability to seamlessly update the database and have the ability to roll back the changes including the database changes if things go wrong.

Database Maintenance

Usually when you deploy new features, there have to be database schema changes to support these. Depending on the type of database system you are using whether SQL or NoSQL the types of changes could be quite different. In the SQL world you might add some fields to some tables or add some new tables. Generally when you design your schema changes you need to be cognizant of the demands of continuous deployment. For a web site that needs to be up 24×7, you can’t deploy something that requires hours of data conversion. Generally you want a very simple database design to allow changes without a great deal of fuss. If you end up really needing a major conversion, then this will have to be planned and you need to measure the ROI of such a feature given the disruption to customers of even small outages. Another approach is to make the application smarter so perhaps it can handle old data and new data so that the system can still run as the conversion is in progress. This of course takes more work that simply shutting things down, but becomes the cost of doing business in the 24×7 SaaS world.

Besides providing scripts and/or programs to convert the database, you have to provide scripts and/or programs to un-convert the database. By the same token if something goes wrong, say users start having problems, then you need to roll back the database changes (and the newly deployed build), again without causing a huge service blackout. This side of things can be very challenging. These processes need to be a heavy focus for unit and other automated testing.

Rollback

Not all changes that pass all the testing and have been approved by all departments end up delighting customers. Often some unforeseen usage causes the new features to greatly irritate your customers, who then tell you and the world all about their annoyance on Twitter, LinkedIn, Facebook, etc. At this point you may need to make the hard decision to roll-back the newly deployed build along with any artifacts like database changes.

Even though we hate doing this, we need to remember that customer satisfaction is job one and that if we keep annoying them, they will go to another service that won’t annoy them so much. All this is a part of being Agile, customer focused and applying validated learning. Basically learn everything you can about what happened so you can do better next time and actually delight your customers. Over time you will probably become comfortable with rolling back at the first sign of trouble and just chalk it up to a learning experience.

Continuous Monitoring

As part of continuous deployment, you need continuous monitoring of your systems. You need to know what features people are using and how. From this you can learn where your users are running into problems and use this information to improve the situation for future builds.

You can also save yourself the headache of a Twitter storm if you detect a problem users are having before too many get mad about it and either fix or roll-back the annoyance.

The nice things about SaaS services is that you have complete control of the service and can track all sorts of usage metrics and information (of course subject to privacy policies).

Summary

A rather good blog post on WordPress’ experience with continuous deployment is here (the site this blog is hosted on). Just to emphasize that many companies are really doing this. In WordPress’ case deploying around 16 builds per day to customers. As companies perfect this process it is becoming a competitive weapon where whoever does this best, innovates faster and leaves their competition in the dust.