Outbox

Introduction

An outbox coordinates outgoing messages with the work that produced them. It prevents a consumer from publishing or sending messages when its business work fails, and ensures downstream consumers do not observe those messages before that work has completed.

MassTransit supports an In-Memory Outbox, which buffers messages in the process, and a Transactional Outbox, which stores messages with application data so they can be delivered after a process restart. The Transactional Outbox also includes a Bus Outbox for messages published or sent outside a consumer.

Retry, Redelivery, and Outbox

Retry, redelivery, and the outbox solve related but different problems:

Mechanism	What it does	Use it when
Message retry	Re-executes the consumer while retaining the message delivery.	A short-lived, transient failure may succeed on another attempt.
Message redelivery	Returns the message to the broker for a later attempt.	A dependency may need minutes, rather than seconds, to recover.
Outbox	Holds outgoing messages until consumer work succeeds.	A consumer changes state and also publishes or sends messages.

Retry and redelivery determine when consumer work is attempted again. An outbox determines when the messages produced by that work become visible. They are normally used together: configure retry and, when appropriate, redelivery for transient failures, then use an outbox to avoid sending messages from failed attempts.

Idempotency is still important. Message brokers and distributed systems are at-least-once by nature, so consumers must safely handle a message that is delivered again and avoid repeating externally visible work.

Choose an Outbox

Situation	Recommended approach
The consumer has no durable state change, or duplicate outgoing messages are acceptable.	Use regular publish/send and configure retry or redelivery as needed.
The consumer updates state and publishes or sends messages; replay after a process failure can safely reproduce the messages.	Use the In-Memory Outbox with retry/redelivery and idempotent business operations.
The consumer updates durable state and its outgoing messages must survive a process failure after the state is committed.	Use the Transactional Consumer Outbox.
An API, controller, or domain service writes to a database and publishes or sends in the same container scope.	Use the Transactional Bus Outbox.

The outbox does not replace a durable broker, retry/redelivery policies, or idempotent business logic.

In-Memory Outbox

The In-Memory Outbox buffers messages produced by a consumer until the consumer completes successfully. It is fast and requires no outbox database tables, but the buffer is lost if the process stops before it delivers the messages to the broker.

MassTransit implements messaging patterns, many of which are designed to ease the transition from a tightly coupled, database-centric application to a set of services that are highly available, reliable, and eventually consistent. Some of these patterns are clear, but some of them require a little more explanation to truly understand how they are best used.

The Outbox

Consider a consumer or saga that updates a database record and publishes an event. Sending that event before the database update succeeds lets downstream services act on state that was never committed. Retrying the consumer without an outbox can also send the same event once per attempt.

The outbox holds messages until the transactional portion of message processing has completed. With a saga, messages are delivered after its state is saved. This ensures that the database is updated before consumers can start processing the produced messages.

The In-Memory Outbox

The In-Memory Outbox, a feature included with MassTransit, holds published and sent messages in memory until the message is processed successfully (such as the saga being saved to the database). Once the received message has been processed, the message is delivered to the broker and the received message is acknowledged as successful.

MassTransit consumes messages in acknowledgement mode. The broker locks the message, and the message is invisible to other consumers until it is either acknowledged (ack’d) by the consumer or negatively-acknowledged (n’ack’d) explicitly by the consumer or implicitly due to a service or network failure.

See In-Memory Outbox configuration for endpoint and consumer configuration.

cfg.ReceiveEndpoint("r-trashy-saga", e =>
{
    e.UseMessageRetry(r => r.Immediate(5));
    e.UseInMemoryOutbox(context);

    e.ConfigureSaga<TrashState>(context);
});

Production services should configure retry for transient failures and redelivery when a longer recovery window is appropriate. Do not retry business constraint violations that cannot succeed on another attempt.

But what if the message doesn’t send?

This question comes up, and it is a fair question. If the broker goes down, the outbox would be unable to deliver the messages. If the process crashes, the messages in the outbox would be lost. Both of these failures can happen, though it is rare. And if computer science has one rule, it is that the rare will always happen. In production. On a Friday afternoon.

Failure Walkthrough: Take out the Trash

Imagine you’re twelve, sitting on the sofa, playing video games with your friends. Suddenly, from the other room, you hear your mom call out, “Take out the trash!” Of course, you’re in the middle of a battle, and while you’ve explained many times that you can’t pause a multiplayer game, mom just doesn’t get it. So you do what any 12-year-old does, you ignore her. The trash remains right where it is, in the kitchen.

After a while, the lack of a door opening and closing, the still present smell of burnt popcorn from the kitchen, and your mom calls out again, “Take out the trash.” At this point, you’re dead, in spectator mode, and decide to comply – you take out the trash. Then you slide back onto the sofa and get ready for round two.

More time passes, the squad is ready, and you’re about to get on the bus. Your mom, however, didn’t hear from you and shouts once more, “I said take out the trash.” “Mom, I already took it out,” you reply, realizing after that you forgot to mute your mic. The jests and jokes begin as you thank the bus driver and head out.

The Story

This real-world example includes both failure scenarios that are brought up when considering the in-memory outbox.

First, it didn’t happen. The database may have been unavailable or the service crashed deserializing the message. Either way, it failed. And the message? It’s still on the broker. It will be redelivered. Mom will keep telling you to take out the trash until you take it out.

Second, it happened, but the messages were not delivered. You didn’t tell her you took it out. In this case, the message will also be retried. But in this case, this rare case, this is where the previously mentioned term idempotence comes back onto the field.

When the message is attempted a third time (and face it, the third time is dangerously close to getting a chancla to the head), the database was already updated. The invoice is already approved, in the database. The messages weren’t sent, however, so other services may not know that the invoice was approved. In this case, for the service to be idempotent, it should assume that:

1. The message delivery failed because it is being delivered – again.
2. Since the invoice is approved, and this is the approve invoice command, something must have failed after the database was updated.
3. The only thing after the database update is the outbox delivering messages.

Study Occam’s Razer (okay, yeah, I’m a fan of Razer gaming gear, so I’m leaving it spelled that way)

The correct thing to do at this point is to use the state in the database, along with any information that is contained in the message, to produce the same commands and events produced in the previous attempt. Those messages will be delivered by the outbox, and the message will be acknowledged.

!! Victory !!

That’s it, an easy-to-use, reliable solution to perform atomic operations that update a database and send/publish messages, and it works for any database updates that are sent as commands (delivered by durable message queues).

And a big thank-you to Jimmy Bogard, whose tweet prompted me to write this article!

Other Reading

Transactional Outbox NServiceBus Outbox

Transactional Outbox

It is common that a service may need to combine database writes with publishing events and/or sending commands. And in this scenario, it is usually desirable to do this atomically in a transaction. However, message brokers typically do not participate in transactions. Even if a message broker did support transactions, it would require a two-phase commit (2PC) which should be avoided whenever possible.

YouTube

See the Bus Outbox and Consumer Outbox in action in The New Transactional Outbox.

While MassTransit has long provided an in-memory outbox, there has often been criticism that it isn’t a real outbox. And while I have proven that it works, is reliable, and is extremely fast (broker message delivery speed), it does require care to ensure operations are idempotent and when an idempotent operation is detected, events are republished. The in-memory outbox also does not function as an inbox, so exactly-once message delivery is not supported.

The Transactional Outbox has two main components:

• The Bus Outbox works within a container scope (such as the scope created for an ASP.NET Controller) and adds published and sent messages to the specified DbContext. Once the changes are saved, the messages are available to the delivery service which delivers them to the broker.

• The Consumer Outbox is a combination of an inbox and an outbox. The inbox is used to keep track of received messages to guarantee exactly-once consumer behavior. The outbox is used to store published and sent messages until the consumer completes successfully. Once completed, the stored messages are delivered to the broker, after which the received message is acknowledged. The Consumer Outbox works with all consumer types, including Consumers, Sagas, and Routing Slip Activities.

Either of these components can be used independently or both at the same time.

Bus Outbox Behavior

Normally when messages are published or sent, they are delivered directly to the message broker:

When the bus outbox is configured, the scoped interfaces are replaced with versions that write to the outbox. Since ISendEndpointProvider and IPublishEndpoint are registered as scoped in the container, they are able to share the same scope as the DbContext used by the application.

Once the changes are saved in the DbContext (typically by the application calling SaveChangesAsync), the messages will be written to the database as part of the transaction and will be available to the delivery service.

The delivery service queries the OutboxMessage table for messages published or sent via the Bus Outbox, and attempts to deliver any messages found to the message broker.

The delivery service uses the OutboxState table to ensure that messages are delivered to the broker in the order they were published/sent. The OutboxState table is also used to lock messages so that multiple instances of the delivery service can coexist without a conflict.

Consumer Outbox Behavior

Normally, when messages are published or sent by a consumer or one of its dependencies, they are delivered directly to the message broker:

When the outbox is configured, the behavior changes. As a message is received, the inbox is used to lock the message by MessageId.

When the consumer publishes or sends a message, instead of being delivered to the broker, it is stored in the OutboxMessage table.

Once the consumer completes and the messages are saved to the outbox, those messages are delivered to the message broker in the order they were produced.

If there are issues delivering the messages to the broker, message retry will continue to attempt message delivery.

For details on configuring the transactional outbox, refer to the configuration section.