Deployments of event-driven architecture (EDA) are now accelerating, with over 4 in 5 IT leaders saying their company plans to apply EDA to 2-3 new use cases within the next 24 months; a new IDC Infobrief charts this rise.
The right broker provides the backbone to real-time application integration.
At the very heart of EDA transformation is the event broker – middleware software that routes events and other data between various applications, systems and devices. It is the cornerstone of event-driven architecture, and all event-driven applications use some form of an event broker to send and receive information.
Analysts are jumping into the debate, not whether to embrace EDA, but rather which event broker to choose to underpin their EDA, notably, David Mooter of Forrester who recently outlined the choice between a “log stream broker” vs a “smart broker”.
Log-Based Brokers Stretched to Operational Limits
Log stream brokers are best epitomized by Apache Kafka. Over the last few years Apache Kafka has taken the world of data streaming by storm, because it’s very good at its intended purpose of aggregating massive amounts of “log” data and streaming it to analytics engines and big data repositories. Unfortunately, the popularity and prevalence of Kafka has led many developers to use Kafka for use cases for which it is not ideally suited – namely operational, “run-the-business” scenarios which often involve a mix of applications, systems and devices that need to tap into specific event streams to run effectively.
Log based brokers use rigid, flat topic structures to describe the data they transmit, which puts the onus on the applications to do the work to filter through all the data that’s being streamed. This can be like drinking from a firehose – as applications need to consume, filter and throw away events they don’t need, which leads to increased cost and complexity, not to mention security concerns.
As Businesses Get More Connected, The Broker Must Get Smarter
On the flip side, “smart” brokers do a lot of the thinking, filtering, routing – especially as enterprises work to become more integrated, connected and real-time across diverse Information Technology (IT) and Operational Technology (OT).
These brokers feature rich and flexible topic hierarchies which enable applications to easily publish and subscribe to the very specific subsets of data they are interested in. From an event streaming perspective, smart brokers support a wide range of message exchange patterns beyond publish/subscribe, including request/reply, streaming and replay, as well as different qualities of service, such as best effort and guaranteed delivery – making smart brokers ideal for operational use cases.
A Dynamic Business Needs a Dynamic Broker
The choice becomes clear – if an organization is looking to address operational “run-the-business” applications and use cases across a distributed enterprise, they need a smart broker, providing them three critical attributes:
Detailed Routing and Filtering
When dealing with complex business operations, events need to be intelligently routed and filtered to be made available to the right people at the right time, not just served up en masse. A smart event broker allows consumers to subscribe to only receive the subset of events they need, in the original sequence. Take the operational example of a retailer using microservices to stream order-related events – but different functions are being dealt with in different locations. Those application users who only want to process new orders do not require information such as shipped orders or returns; they just require information based on new orders.
Targeting Events for the Flight Schedule
It’s also the job of a smart broker to publish information in a way that allows fine-grained filtering of events based on topic taxonomy. Consider an example of an aviation authority that is required to manage incoming and outgoing flights from a particular airport. A smart broker can handle the volume of events related to all flights, but allow subscribers to drill-down by airline, arrival/departure, on time/delayed, inbound and outbound gate for each and every flight.
Event Mesh Compatibility
Organizations who require real-time event streaming for operational use cases need to look at how their broker can support an “event mesh”. An event mesh is an architecture layer consisting of a network of event brokers interconnected to allow events from one application to be dynamically routed and received by any other application no matter where these applications are deployed – no-cloud, private cloud, or public cloud.
Consider a manufacturing organization using IoT edge devices for asset tracking, product quality monitoring and predictive maintenance. With an event mesh, sensor events in manufacturing processes can be leveraged with real-time streaming analytics to improve quality and detect machine maintenance issues sooner. Operations are always on, and so are customer and employee expectations.
Deep Application Integration
Catering to the complex business ecosystems they support, a smart broker easily connects a variety of applications and devices without worrying about protocol translation from one to another. A distributed organization, for example, has many custom applications in many different languages that they need to have share information.
Operational use cases will usually involve this complex mix of legacy and modern applications as well as taking on further data streams from IoT devices – all communicating using different languages and protocols.
Heineken: A Worldwide Digital Brewer
Heineken, the multinational brewing company, has business units that are spread across 190 countries, and is the perfect use case of how EDA deployments have increased in sophistication. Heineken uses an event-driven integration approach to support event streaming across thousands of business-critical applications touching payments, logistics, inventory management, and more. Only a smart broker enables this consistently fast, reliable and robust integration process across all key business functions.
CIOs and DevOps teams need to be broker aware – achieving the desired outcome with EDA often depends on choice of event broker, and some are better suited for operational tasks and use cases than others.
