Micro-services, common pitfalls and Lambda architectures as a potential solution to avoid them

There is a fundamental conflict between services and data systems. Micro services are designed to encapsulate data within their inner workings, data systems are designed for the purpose of exposing data.

In a micro services based architecture an application is split into fragments or micro services which are separated by a network and collectively collaborate and work together to achieve the same goals as a monolithic application.

Micro services encapsulate distinct code and logic and manage their own data, they allow for easier scaling in both performance and organizational terms, and their biggest benefit, if done correctly is their capability to be deployed independently of one another.

There are several anti-patterns that can lead to micro-services being coupled to each other and therefor cause them to lose their most important benefit of being able to be deployed independently.
Having a shared package amongst micro services is an anti-pattern since making changes to the shared package might require several micro services to make changes to their inner workings and their interactions with this shared package and hence not allowing the micro services to evolve and be deployed independently of each other. To counter this scenario, it is advised to have well defined responsibilities and clean separation between services and develop contracts that clearly define the roles of a specific service, if a new requirement does not fit into any of these contracts then it probably is a good candidate for a new micro service.

In a micro services architecture each micro service will own and manage its own set of data, and if there is a reporting need for a larger set of data which is an aggregation of all the data managed by all the micro services then the data from all these micro services is gathered into a central location where reporting needs are met. In such a set up each micro service potentially has a different way of representing the same data structure and over time these data structures which define the same concepts in the overall system can potentially diverge, and also copying and moving around all this data in a distributed system where essentially these is no single source of truth does not come without problems.

We should always design systems with flexibility in mind, in such way that allow for extensibility and change, because In the real world business evolves and with new requirements there is always the possibility that a shared set of data between these fragments would become a requirement, this could potentially lead to the coupling services by using a shared data source where data manipulation occurs by multiple services, as opposed to where only querying and reporting tasks are achieved which does not affect the agility of micro services.

The introduction of event stream analysis technology such as Apache Kafka and Microsoft Azure Stream Analytics allows for a new approach to designing micro services that can allow us to avoid these common pitfalls.

Using event sourcing, event stream analysis platforms and Implementing a distributed log, can allow us to avoid the coupling caused by having a single source of truth where data manipulation and querying takes place by multiple services and allow the micro services to grow independently.

In such systems all data is first treated as an event, and all events are stored in a distributed log managed by stream analytics services, essentially this means that there is a single source of truth for all the data and events that can be tapped into, both for reporting purposes and business requirements. The micro services that need to access this data have no control over it and can only query, react to or copy the data into their own storage mechanisms.

With this setup we can be sure that we can easily scale as the events are stored in a distributed log, we also are making sure that the separation of concerns between the micro services is held as none of the micro services can modify the data in a way which would cause discrepancies and we also have a single source of truth which can be analysed at any time both for machine learning purposes or business reporting however event sourcing does not come without downsides and you are well advised to take these into consideration before building a system based on this pattern.