The Topic, the Message and the Partition

Traditional messaging patterns: message queue and publish – subscribe, have some limitations as a result of their design.

In the previous post – Apache Kafka Ideas – Part 1, a couple of messaging use cases were introduced. In order to define those cases with Kafka, it is important to understand its ideas. At the very heart of Kafka are topics and partitions. This post explains basic concepts behind them.

I currently work on a project were AWS RDS PostgreSQL is used as a data storage. Since Postgres is using a mechanism called Multiversion Concurrency Control – MVCC, an UPDATE or DELETE command does not remove old versions of a row immediately. These are left on a disk, waiting to be collected and cleaned by a vacuum process. Vacuum can be automated and autovacuum serves that purpose.

Once autovacuum is configured, how do I know it works as expected? Is it triggered when I expect it to be?