Integration can be quite complex, especially when you are talking about making your swanky new application work with a legacy system. I’ve seen multiple generations of patterns to tackle this problem gain prominence over the past decade and a half. Starting with SOAP web-services to Message Bus to Enterprise Service Bus, REST API and even WebSockets! Each of them has been the blue-eyed boy, or heartthrob of their generation and seen their popularity vain as the new kid on the block got more famous. Having said that, and allowing myself to feel relatively old, whatever the approach, the bigger challenge in integrating legacy services has been that they might eventually get replaced with other services soon, or soon enough for you to get into yet another exercise of re-integrating with the new system all over again with a sense of repetitive labour induced fatigue. Shameless plug, We at NimbleWork were facing this very strategic question when our SaaS journey started a couple of years ago; and this blog is about our learnings and experience in using Choreography as the cornerstone of our integration approach just like it was for implementing Strangler Fig.
There are many definitions of Choreography if you happen to browse popular articles or blogs over the web. The way I like to put it is
Choreography represents the communication between microservices through the publishing and subscribing of domain events.
It essentially means that each of the services publishes a Domain Event notifying a change that’s occurred out of an action initiated on the service. Services to whom this event is of interest subscribe to them and act accordingly. It’s an extremely efficient, lightweight, distributed chain of command. Choreography has emerged as the mechanism of choice in implementing the Saga Pattern in Microservices, and I’ve seen it increasingly replace Orchestrators. I’d like to point out however that it’s not a universally applicable solution or implementation/design in the microservices world, there’s some criticism of this approach going around too, though I haven’t burnt my hands using it yet, so I’d like to think the criticism is more out of design issues elsewhere which might have lead to bottlenecks in using Choreography.
Here’s our approach, both the legacy system and the new systems emit Domain Events, which are published to Kafka. Each Kafka topic is backed by MongoDB collection using the Kafka MongoDB connector for Fault Tolerance and HA purposes. Each of these collections is replicated over a MongoDB replica-set. In case of failed processing of the domain event, the failing side emits an event for compensating transactions instead of rollback. Any outage on either side is covered via two factors
There isn’t much we have to do as developers for either of these provisions but knowing that we have these provisions is more than just handy knowledge.
Handling failures, like most modern architectures is a layered approach in this case. So let’s look at them from the perspective of where we choose to tackle them.
Choreography is a strong pattern in Microservices and is here to stay for good. This blog post however was our attempt to show that we can improvise to use well-established patterns for much wider goals than they were envisioned for. It requires a bit of imagination, lots and lots of reading, hours of experimentation and not to forget, the temperament the handle their failures. I hope this blog helps fellow engineers with one more option to consider while in the transition phase of their microservices journey where new services have to integrate and play well with legacy systems that are built on completely different designs and patterns altogether.