DevOps and Microservices
Updated: Aug 22, 2020
A lot has been said about both DevOps and Microservices architecture for me try define anything about these two, so in this post I'll share my experience on how DevOps and Microservices architecture play together.
Microservices architecture can be defined as an architectural style with the following key characteristics:
Discrete services organised around business capabilities
Decentralised data management and governance
Smart endpoints connected through dumb pipes
This architectural style has proven it's mettle in helping scale up systems to handle terabytes of data and million of requests and is here to stay. DevOps, on the other hand has also matured a lot since it's inception about a decade ago. The core principles of DevOps haven't changed much but how we practice it and how the automation is implemented is quite a melting pot of sorts in itself. We'll see how microservices plays with some the key DevOps automation practices and how (I believe) either of them work best with only the other.
Through the rest of this article I'll draw parallels from a product I helped implement Continuous deployment, testing and Delivery pipeline. The product or platform rather had three layers
Platform - this was the core business logic and services of the product
Function - this was platform + extended services for a specific purpose + customised UIs
Solution - this was function overlay-ed with additional customer specific modules
In total we're talking about a 200 maven modules, three different ZIP bundles each of which which had whole lot of EJB modules EAR, SAR (JBoss) and in some cases JavaEE containers bundled into them, not to mention the database change management module too. We'll talk about how the various DevOps automation practices worked before and after microservices migration.
Continuous Integration, Deployment and Delivery
When you're dealing with the kind of architecture outlined above, multistage CI is the only option at hand. Even with the simplest of designs you would end up with one pipeline each for the versions you're supporting for each of the customers. In our case all the layers had multiple supported versions. So we ended up with a lot of Jenkins builds that would just run CI for the active branches, then there were pipelines for Continuous Deployment and nightly, sprint and major version releases, not to mention the manual efforts of merging and repackaging by creating jobs on the fly for customer upgrades. And it all goes out of hand no matter how much automation you develop to manage the automation!
Breaking into microservices we ended up with three microservices groups, each of which then had their own share of microservices, I'll not delve into how we went about with our microservices split, but on how we chose to build and package them. Our continuous delivery pipeline utilised docker containers in the initial part and then bundled both virtual appliance (for customers not running on containers or k8s) and docker containers. Even with that our CI Builds had the following standard three jobs for each of the microservices
A CI Build that produces a docker container
A nightly pipeline job that takes latest stable docker containers from CI builds, deploys the system and runs through performance, regression and security tests
A release build which basically produces releasable system images and docker containers
There's no mesh of CI triggers working on both source control and upstream project triggers, nor do we have vast variety of pipelines. The overall CD flow for each of the microservices therefore becomes more or less same which meant we could reuse our builds better.
Since we're shipping domain based microservices and not versioned layers of our system here, we expose breaking changes as a new version of end point in existing APIs and leave it for the customers to choose when they want to migrate to the newer versions. Going by this experience we could certainly say moving to microservices improves both "Shift Left" and "First Way" of DevOps. We could also argue the that holy-grail of one click releases looks more possible in microservices architecture than on the older architecture.
Let's look at the development flow before we went to microservices
We had to split teams into "Core" which worked on the platform libraries and "Feature" teams which worked on Function and Solution layers. The problem with that is, as you can see, a complex cycle of release management which requires a quit some laborious workflow management and feedback cycle. This cycle is at-least a sprint in it's fastest form for the feedback to reach feature teams. The microservices on the other hand reduced the lead time for teams, this is because any changes on an upstream microservices were made available through versioned API endpoints, deployed as containers so we had done away with code merges, and pulling updated versions of libraries for changes to be available downstream. This reduced the feedback to running tests with the downstream service running against the newly versioned endpoints. We had also done away with the sense of pristine QA environment with controlled deploys as each of the teams could replicate the entire environment with some updated services easily. They would then publish their system images and containers to our release registry for it to be part of release train.
The scenarios outlined here are an indication of how you can speed up idea to go live for microservices through DevOps and that it's in the microservices architecture that you see DevOps work really well, perhaps better than other architectures. I'd like to throw a word of caution, what you see here is the state we're in after countless small experiments and tuning of our flow based on observations. Our microservices organisation itself went through iterations so did our continuous deployment and testing cycles. This flow might not work off the shelf for every other project. I do hope however, that I've been able to give a good glimpse of the thought process that went behind running microservices architecture more efficiently using DevOps.