The publish-subscribe pattern: Everything you need to know about this messaging pattern

As applications become larger and more complex, they increasingly need an efficient and scalable way for their components to communicate. Traditional approaches that rely on components communicating directly, while effective in simpler systems, often falter under the demands of modern distributed environments.

This article explains the publish-subscribe pattern, an asynchronous way for components in distributed systems to communicate. It explores the key principles of decoupled communication and its real-world applications in systems like stock trading, social media notifications, and online gaming.

What is the publish-subscribe pattern?

While traditional client-server systems that depend on synchronous request-response communication were suitable for simpler applications, they struggle with the demands of modern information-driven applications. For use cases that require the fastest possible communication between components (such as news delivery, stock trading, and air traffic control), classic tightly coupled techniques such as remote procedure calls (RPC) cannot handle the distribution of this data at a high enough speed.

The publish-subscribe pattern allows for indirect and asynchronous communication, reducing the coupling between components compared to synchronous systems, and improving performance by allowing individual components to operate at their optimal speed. In "pub-sub," messages are packets of data, such as events, notifications, or commands, which are sent by publishers. The messages are transmitted through channels, and subscribers express interest in specific channels to receive only the messages they care about.

Example of a publish-subscribe pattern

• Publisher: A weather application sends weather updates to a "weather updates" channel. It doesn’t need to worry about the architecture or state of the subscribers; it simply publishes updates in a predefined format, leaving the channel to handle distribution.

• Subscriber: Devices like IoT sensors or smartphones receive updates from the channel to display local weather information or trigger-related actions like sending alerts for extreme weather warnings. Each subscriber does not need to wait for the others, allowing each to update independently.

How the publish-subscribe pattern works

At its core, the publish-subscribe pattern uses a messaging system and channels to distribute events from publishers to subscribers.

Messaging is a technology that enables high-speed, asynchronous, program-to-program communication with reliable delivery. Programs communicate by sending each other packets of data, called messages, over channels.

Channels are logical pathways that connect the programs and convey messages. A channel is the virtual pipe that connects a sender to a receiver. A channel is like an array of messages that’s accessible from multiple computers, so that several applications can use it concurrently.

Each message is a data structure — such as a string, an object, a record, or just a byte array. It can be interpreted as data, as the description of a command to be invoked by the receiver, or as the description of an event that occurred in the sender. 

A message contains two parts: a header and a body. The header contains meta-information about the message, which is used to deliver it: who sent it and where it’s going. The body contains the data being transmitted, which the messaging system itself can ignore as it's intended solely for the receiver (subscriber) to act on.

The publish-subscribe pattern is an anonymous, many-to-many, asynchronous communication paradigm (“anonymous” in the sense that publishers and subscribers don’t need to know each other). Producers and subscribers are decoupled from each other in several ways:

• Space decoupling: Publishers don’t need to know how many subscribers there are and subscribers don’t need to know how many publishers there are.

• Time decoupling: The publisher and subscriber don’t need to be active at the same time. Events can be published while the subscriber is offline, or the subscriber can be notified of events even if the publisher is no longer connected.

• Synchronization decoupling: Publishers can continue producing events without being delayed, and subscribers can receive notifications asynchronously, allowing them to perform other tasks at the same time.

The role of events in publish-subscribe systems

An event is an occurrence that has happened in an application in the past, and cannot be changed. An event could be, for example, a new customer creating an account in your app, a customer making a payment, or a user failing to authenticate. In publish-subscribe patterns for event-driven architectures, events trigger messages — however, the communication systems are usually not concerned with the cause of events themselves, only the messages they must deliver.

Event notification: An event notification typically carries the absolute minimum state: perhaps just the identifier (ID) of an entity or the exact time of occurrence of their payload. Subscribers notified of that event may take action, like recording the event or using the provided ID to query a service or data store for additional details about the event payload.

Above, ServiceA and ServiceB subscribe to the PaymentCreated event. While ServiceA is satisfied with the information contained in the event, ServiceB requires additional information and must make a call back to the Payment service to fetch it.

Event-carried state transfer: Event-carried state transfer is an asynchronous cousin to representational state transfer (REST). Unlike the on-demand pull model of REST, event-carried state transfer uses a push model, where data changes are sent automatically by the publisher to any subscribers that are interested in receiving the updates.

As illustrated above, ServiceB gets the information it needs after being notified. Now, the PaymentCreated event contains everything ServiceB requires.

Advantages of the publish-subscribe pattern

Increased scalability

The publish-subscribe pattern is scalable by nature, enabling systems to handle large volumes of messages and users without overloading individual components. By distributing message processing across multiple subscribers using a messaging system, pub-sub systems can handle high throughput, and new publishers and subscribers can be added without disrupting the system, allowing for horizontal scaling.

Asynchronous communication

Pub-sub facilitates asynchronous communication, which means publishers can send messages without needing to wait for subscribers to process them. This non-blocking behavior ensures that publishers remain efficient and responsive. Subscribers can consume messages at their own pace, which is critical for systems requiring high availability and responsiveness.

Decoupling

One of the core benefits of the publish-subscribe pattern is the decoupling of publishers and subscribers. Publishers do not need to know who subscribes to their messages, and subscribers are unaware of the source of the messages. This independence simplifies maintenance and enhances modularity.

Reliability

Pub-sub systems often include features like message durability, retries, and acknowledgements, ensuring reliable communication between components. Messages can be stored until successfully delivered to subscribers, minimizing the risk of data loss due to temporary network failures or subscriber downtime.

Challenges and limitations of the publish-subscribe pattern

• Complex debugging and monitoring: Decoupled communication makes it difficult to trace issues such as dropped messages or incorrect processing.

• Message delivery guarantees: Most systems provide "at least once" or "best-effort" delivery, leading to potential duplicates or missed messages.

• Message ordering: Ensuring the correct order of messages across distributed systems can be challenging and may require additional mechanisms.

• Performance challenges: If messaging systems are not implemented appropriately, it can lead to performance issues, like poor or unpredictable latency and high bandwidth consumption. A large number of topics, or many subscribers, can put a strain on the system, requiring adjustments and careful tuning to avoid bottlenecks.

Should you use the publish-subscribe pattern?

Publish-subscribe vs. point-to-point messaging

The publish-subscribe pattern enables one-to-many communication, ideal for use cases like broadcasting notifications or event-driven architectures. Another messaging system to consider would be point-to-point messaging, which involves a one-to-one communication model; messages are sent to a specific queue, and only one consumer processes it. This is commonly used for task queues or job processing, where a single worker handles each message: for example, a customer support system where each ticket is placed in a queue and an available agent retrieves and addresses it.

When to use pub-sub

• For one-to-many communication: Pub-sub is ideal if you need one-to-many communication, such as for broadcasting events.

• To solve scalability challenges: Use pub-sub if your application requires asynchronous communication to handle high message volumes.

• To enable real-time updates: Use pub-sub for live data feeds where instant updates are critical.

When not to use pub-sub

• For point-to-point communication: For tasks that require a single consumer per message, such as job queues, point-to-point messaging is more suitable.

• When you need strict ordering: If message order is critical, pub-sub may not be suitable unless combined with additional mechanisms (such as timestamps).

• For simple use cases: For lightweight systems with minimal messaging needs, pub-sub may be overkill.

Use cases for the publish-subscribe pattern

The publish-subscribe pattern can help to build efficient systems for different use cases, such as:

Broadcasting notifications: social media, retail inventory

By using the publish-subscribe pattern, large-scale systems like X (Twitter) or Facebook are able to send notifications about specific hashtags to all interested subscribers instead of just to a single user or application. Users can also follow each other and receive real-time updates on their activities without the need to continuously poll for updates or manually check each other's profiles. 

The retailer Zalando uses AWS services like SNS and SQS to enable real-time data processing for inventory updates and notifications. The publish-subscribe pattern ensures scalability and decoupled communication.

Real-time systems: gaming, ride sharing

In online gaming, the publish-subscribe pattern plays an important role in delivering real-time updates to players. For example, game state changes — like player movements or score updates — are published to a channel, and all connected game clients (subscribers) receive these updates instantaneously, ensuring the gaming experience remains synchronized. To overcome the challenge of message ordering, publishers may apply a timestamp to messages, and subscribers can then reorder events based on these identifiers before displaying the correct information to the player. 

In a similar way, Uber’s system uses real-time event processing to synchronize drivers and riders.

Event-driven architectures: stock trading, news analytics

In stock trading systems, publishers can publish stock price updates to a channel, and different components can subscribe to the channel to receive the latest price updates without needing to know about each other's existence. This enables the system to update itself in real time, minimizing latency and reducing traffic on the system. 

The New York Times relies on Google Cloud pub-sub for real-time data ingestion and distribution, powering user activity tracking and article performance analysis.

Technologies to build a pub-sub system

Here are some key technologies to build the core components of an event-driven system using the publish-subscribe pattern:

• Message broker: Apache Kafka can act as the backbone for message distribution. It handles high throughput and provides scalable messaging between publishers and subscribers. Alternatives include RabbitMQ or Redis Pub/Sub.

• Publisher-subscriber logic: Node.js or a similar backend runtime can implement the logic for publishing events and subscribing to topics. Libraries like KafkaJS (for Kafka) or amqplib (for RabbitMQ) can simplify this process.

• Frontend communication: The WebSocket protocol can provide real-time delivery of updates from the back end (subscriber) to connected clients (subscriber front end). Server-sent events (SSE) are ideal for applications where updates flow unidirectionally from the server to the client, such as live news feeds. GraphQL subscriptions provide a flexible option for real-time data delivery for systems that already use GraphQL as their API layer, such as collaborative tools.

How Contentful leverages the publish-subscribe pattern to create its high-performance platform

Contentful leverages event-driven architectures (including the publish-subscribe pattern) to create our content platform. We use events and state changes to link our services and microservices-based applications when we need to decouple functionality to prioritize performance, scalability, and resilience.

The Contentful composable content platform allows you to compose rather than build your back ends with our visual builder for creating structured content that can be integrated using our GraphQL and REST APIs. Everything in Contentful is served by our lightning-fast CDN, delivering your assets from the edge, for the best possible user experiences.

Take a tour

Chat with our team

We'll show you how our composable content platform can enhance 1:1 customer connections and drive growth for your business.

Subscribe for updates

Build better digital experiences with Contentful updates direct to your inbox.