Publisher-Subscriber pattern

The Publisher-Subscriber pattern lets applications broadcast events asynchronously to multiple interested consumers without coupling the senders and the receivers. This approach is known as pub/sub messaging.

Context and problem

Cloud-based and distributed applications often include system components that send information to other components as events occur. When a sender communicates directly with its consumers, it must know the identity and endpoint of every consumer, deliver messages to each consumer, and manage failures individually. Adding or removing a consumer requires changes to the sender, which limits how independently teams can develop and deploy components.

Message queues decouple senders from consumers and prevent the sender from blocking a process while it waits for a response. A standard queue creates a direct relationship between a sender and a single consumer. To support multiple consumers, the sender must create a separate queue for each consumer, which increases routing complexity and doesn't scale well. Some consumers need only a subset of the information that the sender produces, but queues don't provide built-in ways to filter messages by content or category.

Many scenarios require a sender to announce events to many interested consumers without knowing who those consumers are. Each consumer also needs a way to decide independently which events to receive.

Solution

Introduce an asynchronous messaging subsystem that includes the following components:

• An input messaging channel that the sender uses. The sender packages events into messages by using a known message format and sends these messages via the input channel. The sender in this pattern is also known as the publisher.

  Note

  A message is a packet of data. An event is a message that notifies other components about a change or an action that occurs. This pattern typically works with events, but it also carries any type of message, including commands and state notifications.

• One output messaging channel for each consumer. The consumers are known as subscribers.

• A mechanism for copying each message from the input channel to the output channels for all subscribers interested in that message. An intermediary like a message broker or event bus typically handles this operation.

The following diagram shows the logical components of this pattern.

Pub/sub messaging has the following benefits:

• Decouples subsystems that need to communicate. Subsystems support independent management, and the broker retains messages even if one or more receivers are offline.

• Increases scalability and improves sender responsiveness. The sender sends a single message to the input channel and then returns to its core processing responsibilities. The messaging infrastructure routes messages to interested subscribers.

• Isolates faults. A subscriber failure doesn't affect the publisher or other subscribers, and the broker retains messages until a recovered subscriber is ready to process them.

• Supports deferred or scheduled processing. Subscribers can wait to pick up messages until off-peak hours, or the system can route or process messages according to a specific schedule.

• Supports integration between systems that use different platforms, programming languages, and communication protocols, and also connects on-premises systems with applications that run in the cloud.

• Improves testability. Channels support monitoring, and messages are available for inspection or logging as part of an integration test strategy.

• Provides separation of concerns for applications. Each application can focus on its core capabilities while the messaging infrastructure handles the work required to reliably route messages to multiple consumers.

Problems and considerations

Consider the following points as you decide how to implement this pattern:

• Existing technologies: Use messaging products and services that support a publish-subscribe model instead of building your own. In Azure, consider the following services:

  • Azure Service Bus for messaging that requires transactions, ordering, sessions, or dead-letter queues.

  • Azure Event Grid for event‑based, push‑delivered notifications, especially when Azure resources change state and need to notify subscribed components.

  • Azure Event Hubs for high-throughput event streaming scenarios like telemetry ingestion and log aggregation. Event Hubs uses a log-based streaming model rather than traditional pub/sub messaging, but it supports multiple consumer groups that read the same stream independently.

  For more information, see Choose between Azure services that deliver messages. Other technologies that support pub/sub messaging include Redis, RabbitMQ, and Apache Kafka.

  Libraries like MassTransit and NServiceBus provide built-in support for the publish-subscribe model on Service Bus and other messaging technologies.

• Subscription handling: The messaging infrastructure must provide mechanisms that consumers use to subscribe to or unsubscribe from available channels.

• Security: Authenticate and authorize both publishers and subscribers on a per-topic basis. Unauthorized publishers that inject messages can damage a system as much as unauthorized subscribers that read them. Encrypt messages in transit, and if content is sensitive, encrypt them at rest in the broker to prevent eavesdropping.

• Subsets of messages: Subscribers are usually interested in only a subset of messages from a publisher. Messaging services often let subscribers select what they receive through the following mechanisms:

  • Topics: Each topic has a dedicated output channel, and each consumer can subscribe to all relevant topics.

  • Content filtering: The broker inspects and distributes messages based on their content. Each subscriber can specify the content that it needs.

  Choose topic granularity deliberately. Broad topics are simpler to manage but require subscribers to filter out messages that they don't need. Narrow topics reduce subscriber-side filtering but increase the number of topics to manage. Some brokers support wildcard subscriptions, like orders.*, which let subscribers match multiple topics without enumerating each one.

• Bidirectional communication: Channels in a publish-subscribe system are unidirectional. If a subscriber needs to acknowledge or communicate status back to the publisher, use the Request-Reply pattern. This pattern uses one channel to send a message to the subscriber and a separate reply channel to communicate back to the publisher.

• Message ordering: The order in which subscribers receive messages isn't guaranteed and doesn't necessarily reflect the order in which the sender created them. If ordering matters, the broker might support ordered delivery within a partition or session, but that constrains scalability. Design subscribers to handle messages independently of arrival order.

• Message priority: Some workloads require that specific messages be processed before others. The Priority Queue pattern provides a mechanism to route higher-priority messages before lower-priority messages.

• Poison messages: A malformed message, or a task that requires access to unavailable resources, can cause a service instance to fail. Capture and store these message details elsewhere for analysis. Some message brokers, like Service Bus, support this process through dead-letter queues.

• Message size: Brokers enforce message size limits. When payloads are large, store the content, like files or images, in an external data store and include a reference in the message. The Claim-Check pattern describes this approach.

• Delivery guarantees and duplicate messages: Messaging systems provide different delivery guarantees that each have trade-offs.

  • At-most-once delivery minimizes overhead but can lose messages if the broker or subscriber fails.

  • At-least-once delivery ensures message delivery but can result in duplicates, like when a sender fails after it posts a message and a new instance repeats the post.

  • Exactly-once delivery removes duplicates but adds coordination overhead and latency, and its availability depends on the messaging infrastructure.

  If your broker doesn't provide deduplication, design subscribers to handle messages idempotently. Different subscribers in the same workload might require different guarantees.

• Message expiration: Some messages have a limited lifetime. If a receiver doesn't process a message within that period, the message becomes irrelevant and the system discards it. Set an expiration timestamp in the message data so that receivers can check its relevance before they process it.

• Message scheduling: A message might be embargoed and unavailable for processing until a specific date and time. Set a release timestamp so that the messaging system withholds the message until that point.

• Message schema evolution: Publishers and subscribers deploy independently, so message schemas change over time. Prefer backward-compatible changes, like adding optional fields, so that existing subscribers continue to work. For breaking changes, version through topic names, like orders.v1 and orders.v2, or through a version field in message metadata. Subscribers should ignore fields that they don't recognize.

• Correlation: The broker decouples publishers from subscribers, which makes it harder to trace the end-to-end flow of a message. Include a correlation ID in every message so that subscribers and logging systems can connect related operations into a single trace.

• Backpressure and scaling: When subscribers can't keep up, unprocessed messages accumulate in the broker and can deplete its resources. Use broker flow control settings to limit unacknowledged messages for each subscriber. Scale out subscribers by using the Competing Consumers pattern when flow control alone isn't sufficient.

When to use this pattern

Use this pattern when:

• An application needs to broadcast information to a significant number of consumers.

• An application needs to communicate with independently developed applications or services. They might use different platforms, programming languages, or communication protocols.

• An application can send information to consumers without requiring real-time responses from them.

• The systems being integrated are designed to support an eventual consistency model for their data.

• An application needs to communicate information to multiple consumers that have different availability requirements or uptime schedules than the sender.

This pattern might not be suitable when:

• An application has only a few consumers that need significantly different information from the producing application. The overhead of a broker adds complexity without any scaling benefit. Direct communication or separate queues might be more suitable.

• An application requires near real-time interaction with consumers. The pub/sub model introduces latency through the broker. Use a request-reply pattern when the publisher requires a synchronous response.

• Consumers must process messages in a specific, guaranteed order. Pub/sub systems generally don't guarantee ordering across subscribers, and maintaining order adds significant constraints to the broker and consumer design.

• The operation requires a single atomic transaction across the publisher and its consumers. Pub/sub messaging is inherently asynchronous and eventually consistent. If you need transactional guarantees, consider a direct database transaction or the Saga pattern for coordinating distributed transactions.

Workload design

Evaluate how to use the Publisher-Subscriber pattern in a workload's design to address the goals and principles covered in the Azure Well-Architected Framework pillars. The following table provides guidance about how this pattern supports the goals of each pillar.

If this pattern introduces trade-offs within a pillar, consider them against the goals of the other pillars.

Example

The following diagram shows an enterprise integration architecture that uses Service Bus to coordinate workflows and Event Grid to notify subsystems of events that occur. For more information, see Enterprise integration on Azure by using message queues and events.

Next steps

• Asynchronous messaging options
• You don't need ordered delivery

Related resources

• Event-driven architecture style
• Idempotent message processing
• Enterprise integration on Azure by using message queues and events