To message bus or not to message bus

Frequently I hear oh should we use REST or an event bus, we want to pass messages around. Oh Async programming, you can’t do that with REST. And other lies and fear mongering that was learned by junior developers just out of the gate and repeated far more times on dev.to and medium.

Event Sourcing — Saving a list of events in a DB

Message Bus — Create pub/sub model

REST — Creating APIs with semantic meaning

These are completely separate things and any one asking a question should we pick X or Y, is irresponsible. That is what is known as a false dichotomy, and it has to stop.

Everything* uses an HTTP interface for communication, and that communication is always synchronous. There is no fundamental async communication between web services, everything is push and poll at some level, even websockets. And therefore anything you want to be pull or async is done on top of that. REST comes with awesome semantics to let the client and server negotiate a strategy.

You always get HTTP for free, and therefore REST for a marginal cost. It cost nothing to do REST wrong, and costs even less if done correctly. Spending two minutes thinking about how to name your endpoints and using the appropriate verb will shower you will be the bliss that many cloud providers, CDNs, and technology exists to encourage. Documentation and model validation comes for free from the open source community.

Event Sourcing

Event sourcing is how you save your data in your DB, full stop, end of story. It doesn’t convey any additional expectations besides that. It has nothing to do with endpoints or resources.

Any service can save a list of events in the DB and expose that data on their interface. An expectation that this has to be done or can’t be done based on the paradigm is misguided. For instance, I can write a simple RESTful api on top of any event sourced resource, and provide that data in a simple and straightforward way. For instance an Audit Trail is one such notion, such as the one used in Authress, is one such example. The list of things that are tracked are an append only log which can’t be changed. You can configure you that trail to return any information you want about the log. How you interface with it is over REST and fully explorable. Everything about event sourcing is orthogonal to REST, it doesn’t make sense to ever bring them up in the same discussion.

The same goes with event sourcing and pub/sub. I can stream data as a publisher from my model, or I can iterate through all the history. The only thing that changes here is where the data is saved.

Always start with the default

The default is always REST, there’s no question about that. It is basically free and your tools no matter what you pick may or may not use it even in a pub/sub model. Try me, go to your favorite pub/sub tool and list out the endpoints and the type of data they work with. These are resources and you use HTTP to work with them. Most of them are even RESTful. The question then always becomes:

Should we also support pub/sub on top of our REST services?

Once you realize that every service has an interface, and that interface is basically free to support REST, and you need to that interface for external api integration; it’s easy to see that pub/sub is a question on top of what you are already building.

Do we also need pub/sub for async?

REST already gives you async, REST is already there, REST lets the service and client decide on how to process, REST is a semantic matching of existing resources to real representations.

Why not ignore REST and build out pub/sub anyway

The simple answer is YAGNI, or you ain’t going to need it. But more specifically

What about the advantages of a pub/sub model, compared to APIs? Future solutions/microservices may be interested in certain state changes or transactions occurring. They can just subscribe to the events they are interested in. You don’t have that model with APIs.

Sure if you have a pub/sub model adding a new service to listen to those events, or reply the old ones again is easy, but you just shot yourself in both feet, cut off your hands, and hung yourself out to dry to get it.

We can still make things backwards compatible, right?

One of the main issues with indiscriminate use of Pub/Sub or Event Buses, is that they lack the standards to support them. The core problem is that they aren’t designed to handle mature use cases like dealing with multiple data schema at the same time. Sure there are workarounds, but unlike REST over HTTP where you can add a simple query parameter or accept header to request a different version of a payload, THERE IS NO COROLLARY for Event Buses. The reason is simple, the consumer cares about data format but isn’t around to tell the provider which format needs to be produced for them. This requires a non-trivial solution to handle multiple data formats.

With microservices I need to write retry logic and handle other services being down.

Sure, but event buses only make this harder. While you don’t have to worry about the dependency being down, you have to worry about the event bus itself being down. Further, when message publishing fails now every worker needs to a strategy for retrying publishing, and what happens if that Single Point of Failure event bus is down. With distributed microservices, you have a simple retry, and return a 5XX if there is a problem. With an event bus you can’t send that message back up the chain because there isn’t one, and if it’s the event bus, instead of one service that has an issue your whole platform is down.

There are ways to secure them, I bet.

Not really, Event Buses don’t solve product problem, they solve a technology problem. That means that they are no longer able to support first class product issues like “who is the user”, “who created this event”, “what should this event be allowed to do”. It’s often an antipattern to say “oh I’ll create this event and let anyone who cares deal with it”. The truth is that even though you can do that, you lose the critical security context that generated the event. Sure if security isn’t a concern and you are building out a simple one app service, it might not be a problem. But for production grade distributed systems, authentication, authorization, and auditing are critically important. You can’t just trust events sent by someone else.

So when does it actually make sense to use pub/sub?

Usually when you have lots of worker nodes that are all equivalently the same and can work on an message and also frequently want to talk with each other, but don’t care which node they talk to. The best example I use are game lobbies for match making.

Players leaving, players coming, games ending, etc… You don’t care which lobby or node picks up that request and they all handle them the same. But it’s a huge waste to have a massive orchestrator trying to coordinate work among them. Every. Other. Scenario…Doesn’t need Pub/Sub.

Pub/Sub === Bad

Pub/Sub also flips the locus of control of the data. Instead of making it the clients concern to care about who owns the data, it becomes the provider’s responsibility to publish it. This makes data processing now significantly harder. The reason is that you can’t source data from your database, you need to publish every resource in two places for consistency, once to your DB and once to your topic/queue etc… This is just an unnecessary extra hard work.

Don’t build out additional things unless you can prove that you need them to solve your special situation. 99% of the time you don’t need anything special, HTTP + REST is totally the right decision. 0.9% you might need to add callbacks or webhooks, websockets, etc… Or maybe you are really in the 0.1%, in that case, don’t listen to anyone tell you anything is always the Right Solution, there is no tech that is a done deal solution to every problem. If someone tells you that, you need to start questioning the source of the information.