Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

As someone who has ran a number of messaging systems in production, this is what my current take is in general:

If you are moving to a more "event-sourced" architecture, usually two main concerns (beyond basic operational stuff of uptime, scale, etc) are routing and long-term retention.

RabbitMQ has the routing but not the retention. Kafka can have the retention and the routing, but it can be complex/expensive. Apache Pulsar really shines here as the API is pub/sub but it is underpinned by a log structure that gives you long-term retention (that doesn't need to be manually re-balanced) but it's flexibility does come with some operation complexity when compared to RabbitMQ.

If your needs is pretty much just moving large amounts of data, Kafka is definitely the most mature and has a big ecosystem, but long term-retention is difficult and there are some sharp edges around consumer groups.

If you really really don't need long-term retention and need complex topologies, RabbitMQ is your best bet and is fairly reasonable to operate even up to fairly high message rates (~10k msgs/sec shouldn't be too hard to achieve)

There are a TON more options these days though, older more java solutions like activeMQ and rocketMQ or more "minimal" implementations like NATs, not to mention the hosted services on cloud providers.

Personally, I am a big fan of Apache Pulsar for it's flexibility and some nice design choices, but I don't think there is any silver bullet in this space.



Would you mind expanding on some of the operational complexity you ran into with pulsar?

I think pulsar is wonderful, but I haven't had the chance to use it for anything serious / in production yet, so I'm curious what pain points you had.


I'm guessing that the pain points surrounded having to set up a Zookeeper cluster in conjunction with Pulsar. I think Pulsar has the best model of the various queuing systems at the moment for the routing flexibility of RabbitMQ, the high-throughput of Kafka (topic/partitions), as well as the ability to seamlessly integrate with cold storage (S3/GCS) and to recall messages from cold storage without extra code (unlike Kafka), I just wish that ZK wasn't an additional dependency.

Anyone know of any Pulsar hosting providers?




Consider applying for YC's Summer 2026 batch! Applications are open till May 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: