Update 2026-03-31: Particular has contributed the SmallBusinessProgram repository to the Use What Works community organization.

Update 2026-01-15: Following Dylan Beattie’s video about licensing models in the .NET ecosystem, and the discussions it triggered in the community, we have decided to open-source our Small Business Program. We’re making it available under the CC-BY license so that everybody can reuse, mix, and modify it without worry of copyright infringement.

We hope this will make it that extra bit easier for others to balance the needs of enterprise licensing with accessibility to smaller businesses.

Here at Particular, we’ve been closely following recent developments in the .NET ecosystem, as our good friends Jimmy Bogard and Chris Patterson have started exploring commercialization options to find a sustainable path forward for their popular open-source projects. We went through this process back in 2010 and can say many good things about how it worked out for both us and the organizations relying on our platform.

One of the things Jimmy and Chris are including as a part of their licensing is a free offering for smaller businesses. Jimmy set his threshold for that at $5 million, while Chris has yet to publish his details. Other vendors have offered this for quite some time, with some minor differences. For example, Duende and SyncFusion have thresholds of $1-3 million, subject to certain conditions.

We believe it is only right that the mature technology that larger enterprises can afford should also be accessible to smaller organizations that don’t have those kinds of budgets.

But before jumping in and offering something similar, we reached out to a bunch of people to better understand their thinking and learned something interesting.

Folks were worried about crossing those thresholds in the coming years and then being hit with a big bill all at once.

That made sense.

So we gave it some thought and came up with a model that replaced that “cliff” with a more approachable “staircase”, where that initial 100% discount gradually steps down, bit by bit, as organizations grow past different financial levels, like so:

This discount structure also layers nicely on top of a variety of pricing models:

• Let’s say you’re a $1.6M org, if full price was $3k/year, after the 90% discount, you’d pay just $300/year or $25/month.
• After growing to $2.4M, if usage also grew to $6k/year, the now 80% discount would bring you to $1200/year or $100/month.

We’re hearing that these are much more reasonable prices for many of the smaller businesses we’ve talked to. Of course, for those organizations under $1M, it would be entirely free.

Want to learn more? Check out the program details and tell us what you think in the comments below!

For SQS, we’ve added automatic message visibility renewal to prevent premature redelivery of long-running messages. For saga persistence with DynamoDB, we’ve enabled optional eventual consistency to help reduce read costs. Plus, with native support for AWS Lambda Annotations, it’s now easier to build serverless message handlers with less boilerplate and better tooling integration. These are just some of the improvements.

Let’s get into the details.

🔗Message visibility renewal

What we did: NServiceBus now automatically prevents duplicate message processing in Amazon Simple Queue Service (SQS) by extending message visibility timeouts during long-running operations. This update eliminates a common source of race conditions and data corruption in distributed systems.

Why we did it: In SQS, each received message becomes invisible for a limited time ¹. If processing takes longer, the visibility timeout can expire, and SQS may redeliver the same message to another consumer, even while the first attempt is still in progress. The extra message delivery leads to multiple issues:

• Non-idempotent handlers can create business or data errors
• The original consumer can’t complete because its receipt becomes invalid
• Message delivery metrics become inaccurate
• A cycle of repeated failures can develop

To solve this, we introduced automatic message visibility renewal. During processing, NServiceBus extends the visibility timeout in 5-minute increments, with configuration options available to customize the renewal duration:

transport.MaxAutoMessageVisibilityRenewalDuration = TimeSpan.FromMinutes(10);

🔗Reserving payload space

What we did: You can now reserve space for third-party tracing headers (like DataDog) to prevent Amazon Simple Queue Service (SQS) message size calculation failures. This new configuration option eliminates intermittent send failures that were previously difficult to diagnose and resolve.

Why we did it: Because SQS messages have a 256 KB limit, NServiceBus has a feature that will offload large messages to S3 to avoid hitting the limit. However, third-party tracing tools like DataDog inject headers just before message dispatch, after NServiceBus has already checked the 256 KB SQS limit. As a result, messages could pass validation but fail during send when the extra headers pushed them over the limit.

To address this, we introduced a configuration option that lets you reserve space for these headers:

transport.ReserveBytesInMessageSizeCalculation = 512; // Size in bytes

This setting allows you to proactively reserve space for third-party headers, such as those commonly used by tracing solutions like DataDog, or diagnostic headers from OpenTelemetry, ensuring the actual message size remains safely below the threshold. You can now tailor your SQS payload calculations to your environment’s specific needs.

The reserved space decreases the maximum allowable message size. That means the transport will trigger the S3 fallback for smaller message sizes, providing a reliable and controlled safety margin for tools that inject headers during send operations.

🔗Support for alternate storage solutions

What we did: You can now turn off S3 payload signing in the Amazon Simple Queue Service (SQS) transport, making it possible to use alternative storage providers like Cloudflare R2 that offer S3-compatible APIs but don’t support signed payloads. This option enables lower-cost providers without being locked into AWS storage.

Why we did it: Until now, large message bodies in SQS were always offloaded to Amazon S3, which requires payload signing. That created a problem for services like Cloudflare R2, which does not support the Streaming SigV4 signing implementation used by the S3 SDK. Because signing was hardcoded, R2 wasn’t usable with the transport.

We’ve introduced a new configuration option:

transport.DisablePayloadSigning = true;

When set, the transport skips the signing step, allowing use of S3-compatible storage providers that don’t support signed payloads. For example, Cloudflare R2 provides zero egress costs, which can significantly reduce expenses for systems handling a high volume of large messages.

If you choose to turn off payload signing, weigh the trade-offs carefully: it unlocks alternative backends but reduces the security guarantees of signed requests.

🔗Promoting SQS message attributes

What we did: SQS message attributes are now automatically promoted to NServiceBus headers, giving direct access to external system metadata without extra integration code.

Why we did it: Previously, these attributes were only available by accessing the native message on the extension context, which limited their usefulness for scenarios like routing, monitoring, or enrichment.

With automatic promotion of SQS message attributes, any custom or third-party attributes on an incoming SQS message are now available as headers throughout the NServiceBus pipeline. This header promotion allows you to inspect and use them from NServiceBus pipeline behaviors ² to enable all sorts of cross-cutting infrastructure concerns.

🔗Improved poison message handling

What we did: The SQS transport now tracks message receive counts (the number of processing attempts before sending a message to the error queue) more accurately and handles poison messages more reliably, reducing duplicate processing, false retries, and wasted compute in production systems.

Why we did it: Previously, message receive counts were based only on an in-memory cache, which had drawbacks. In-memory counts can’t be shared across endpoint instances, so in a scaled-out environment with many nodes, it would take many processing attempts for any one node to observe the message enough times to move a message to the error queue. Additionally, in-memory counts were lost during endpoint restarts, resulting in the loss of historical context.

However, AWS’s ApproximateReceiveCount is not accurate either and has been observed under-reporting actual counts.

The transport now combines both values to determine the actual receive count:

ActualReceiveCount = Max(LocalCacheValue, ApproximateReceiveCount)

This hybrid approach improves accuracy by:

• Buffering against AWS under-reporting
• Recovering from cache loss on restarts
• Auto-correcting in competing consumer scenarios
• Remaining compatible with the existing cache mechanism

Additionally, this improved approach causes poison messages (those that consistently fail) to move to the error queue without invoking business logic. This optimization prevents unnecessary reprocessing, duplicate deliveries, and wasted resources.

Together, these improvements make retry behavior more predictable, simplify operational diagnostics, and facilitate a smoother transition from other transports, such as SQL Server.

🔗AWS Lambda Annotations

What we did: NServiceBus now supports the AWS Lambda Annotations model for .NET, making it easier to build SQS-triggered Lambda functions with less boilerplate and better integration with modern AWS tooling.

Why we did it: Lambda Annotations use C# source generators to reduce repetitive glue code and automatically synchronize CloudFormation templates with annotated methods. With this update, you can register the NServiceBus AWS Lambda integration directly into the .NET service collection provided by the annotation model.

Here’s what a fully functional Lambda handler now looks like:

public class SqsLambda(IAwsLambdaSQSEndpoint serverlessEndpoint){    [LambdaFunction]    [SQSEvent("ServerlessEndpoint")]    public async Task FunctionHandler(SQSEvent evnt, ILambdaContext context)    {        using var cancellationTokenSource =            new CancellationTokenSource(context.RemainingTime.Subtract(DefaultRemainingTimeGracePeriod));        await serverlessEndpoint.Process(evnt, context, cancellationTokenSource.Token);    }    static readonly TimeSpan DefaultRemainingTimeGracePeriod = TimeSpan.FromSeconds(10);}

This integration unlocks a more modern and developer-friendly way to build serverless applications using NServiceBus:

• Eliminates boilerplate code needed to wire up Lambda functions manually
• Leverages compile-time source generation for CloudFormation synchronization
• Reduces cognitive load and setup time for teams adopting AWS Lambda and NServiceBus together

The NServiceBus documentation for AWS Lambda and SQS contains details on the new features, including a ready-to-use sample. If you’re building .NET applications on AWS Lambda and want first-class integration with modern tooling, this update helps you get there with less code.

🔗DynamoDB custom JSON serialization

What we did: To enable advanced scenarios like DynamoDB-specific attribute handling and schema-aware transformations, we introduced support for injecting custom JsonSerializerOptions, JsonTypeInfo, and context resolvers into the saga persistence and mapping APIs.

Why we did it: Previously, the mapping pipeline relied on internal, preconfigured JSON options. These defaults worked for simple objects, but made it difficult (or impossible) to:

• Respect custom serialization attributes like DynamoDBProperty or DynamoDBIgnore
• Inject custom converters or behaviors
• Support schema-specific transformations required by DynamoDB or other systems

With this update, you can override the default options at both the persistence and mapper levels. For example, to support DynamoDB-specific attributes:

readonly JsonSerializerOptions serializerOptions = new(Mapper.Default){    TypeInfoResolver = new DefaultJsonTypeInfoResolver    {        Modifiers = { SupportObjectModelAttributes }    }};

The modifier inspects and manipulates JsonTypeInfo at runtime to recognize and apply DynamoDB semantics:

public static void SupportObjectModelAttributes(JsonTypeInfo typeInfo){    if (typeInfo.Kind != JsonTypeInfoKind.Object)    {        return;    }    foreach (JsonPropertyInfo property in typeInfo.Properties)    {        if (property.AttributeProvider?.GetCustomAttributes(typeof(DynamoDBRenamableAttribute), true)            .SingleOrDefault() is DynamoDBRenamableAttribute renamable &&            !string.IsNullOrEmpty(renamable.AttributeName))        {            property.Name = renamable.AttributeName;        }        else if (property.AttributeProvider?.GetCustomAttributes(typeof(DynamoDBIgnoreAttribute), true)            .SingleOrDefault() is DynamoDBIgnoreAttribute)        {            property.ShouldSerialize = (_, __) => false;        }        else        {            property.ShouldSerialize = (_, __) => false;        }    }}

Now you can serialize complex domain objects using existing DynamoDB conventions without redundantly decorating them with JsonPropertyName or JsonIgnore attributes. For example, you can now serialize a complex domain object like the following Customer type using DynamoDB conventions:

class Customer{    [DynamoDBHashKey("PK")]    public string PartitionKey { get; set; }    [DynamoDBRangeKey("SK")]    public string SortKey { get; set; }    public string CustomerId    {        get => PartitionKey;        set        {            PartitionKey = value;            SortKey = value;        }    }    [DynamoDBProperty]    public bool CustomerPreferred { get; set; }    [DynamoDBIgnore]    public string IgnoredProperty { get; set; }}

Here’s the corresponding message handler, which operates in the same DynamoDB transaction due to the use of the synchronized storage session:

public async Task Handle(MakeCustomerPreferred message, IMessageHandlerContext context){    var session = context.SynchronizedStorageSession.DynamoPersistenceSession();    var customer = await dynamoContext.LoadAsync<Customer>(message.CustomerId, message.CustomerId, context.CancellationToken);    customer.CustomerPreferred = true;    customer.IgnoredProperty = "IgnoredProperty";    // Thanks to the customized serializer options, the mapper understands the context attributes    var customerMap = Mapper.ToMap(customer, serializerOptions);    session.Add(new TransactWriteItem    {        Put = new()        {            Item = customerMap,            ...        }    });}

This change streamlines DynamoDB integration, providing fine-grained control with custom converters or source-generated type resolvers.

🔗Eventually consistent saga reads

What we did: You can now reduce DynamoDB read costs by opting into eventual consistent reads for saga data when using optimistic concurrency. This consistency option provides flexibility in high-throughput or cost-sensitive environments where occasional retries are acceptable.

Why we did it: By default, the NServiceBus DynamoDB saga persister has always used consistent reads to guarantee the most up-to-date saga data and prevent version mismatches or conflicting writes. While safer, this approach consumes twice the read capacity units compared to eventual consistency.

With the new global configuration option, you can enable eventual consistent reads:

var sagas = persistence.Sagas();sagas.UseEventuallyConsistentReads = true;

When enabled, saga reads use DynamoDB’s eventual consistency model, lowering read costs while still supporting optimistic concurrency control.

🔗Summary

If you’ve run into any of the friction points above, we hope these changes make your life easier. If you spot other issues we could improve—or have ideas to make NServiceBus and the Particular Service Platform better—please open a GitHub issue in the relevant repository.

We are the Borg. Your messages have failed and will be assimilated. Resistance is futile.

Distributed systems are a lot like Star Trek’s infamous Borg Collective: massively parallel, highly interconnected, and sometimes, drones services go down or are inaccessible. After all, even with a galaxy-class architecture, things sometimes go wrong. Messages can fail due to transient errors, unexpected exceptions, or configuration hiccups. However, well-designed distributed systems take a stronger cue from the Borg; they are built to be resilient. But how? Let’s explore a more peaceful quadrant of the galaxy to illustrate how this works.

🔗The Starship Enterprise vs. distributed failure

In the world of Star Trek, many things can be replicated with a simple command, and whole universes can be generated with a few phrases in the Holodeck. Still, there are plenty of situations where people want the real thing, whether it’s a bottle of Château Picard, a Klingon bat’leth, or an authentic Bajoran earring.

Imagine you are an operations officer on a Federation starship running the Starfleet Supply Chain system in LCARS. Naturally, it is a distributed system. Orders made on the ship are routed through an NServiceBus-powered system.¹ One of the commands looks like this:

public class PlaceOrderCommand : ICommand{    public string OrderId { get; set; }    public OrderItem[] Items { get; set; }    public string Destination { get; set; }}public class OrderItem{    public string Sku { get; set; }    public int Quantity { get; set; }}

Orders are handled via messages like this:

public class PlaceOrderHandler : IHandleMessages<PlaceOrderCommand>{    private readonly IFederationSupplyApi _supplyApi;    public PlaceOrderHandler(IFederationSupplyApi supplyApi)    {        _supplyApi = supplyApi;    }    public async Task Handle(PlaceOrderCommand message, IMessageHandlerContext context)    {        // This call to the Federation Supply Service will fail when shields are up        await _supplyApi.SubmitOrderRequest(message.OrderId, message.Items);                Console.WriteLine($"Submitted order {message.OrderId} with {message.Items.Length} items to {message.Destination}.");    }}

But oh no! Just as you are placing your own personal order with the computer for a bona fide Klingon d’k tahg, a Romulan ship decloaks off the starboard bow, and the ship’s shields go up. The shields interfere with subspace communications to the Federation Supply Service, causing the external API calls to fail with a ServiceUnavailableException. Did your order go through?

🔗Retry protocols engaged

NServiceBus detects the failure and applies its first-level and second-level retry strategies. But if retries are exhausted and the issue remains, the message is moved to the error queue—not dead-lettered, not deleted, just safely quarantined.

This is where ServicePulse beams in.

🔗ServicePulse: Your command center for failed messages

Unlike many systems that silently drop or bury failed messages in a dead-letter queue that might as well be beamed into space, NServiceBus gives you a mission control interface—ServicePulse.

As shown in the image below, ServicePulse groups failed messages by exception type, and originating endpoint:

In this case, 174 messages failed with a ServiceUnavailableException inside the PlaceOrderHandler handler. These messages were all of type PlaceOrderCommand.

It turns out the Romulan ship is actually part of training exercises that are now complete. With shields down, subspace communications to the Federation Supply Service are restored. We can now perform a level 3 diagnostic. From LCARS, we open ServicePulse and see that there is a list of failed messages.

🔗Manual intervention isn’t futile

Sometimes, you want to investigate specific errors more deeply. With ServicePulse, you can drill down into individual messages, inspect the exception stack-trace, message headers, even the message body, and choose to retry a single message, or even a whole group of messages. This level of control means your system is both resilient and humanoid-friendly.

There is no need to SSH into remote systems to access queues, ² write ad-hoc scripts, or run a recursive algorithm through the main deflector dish—ServicePulse makes distributed systems self-heal like a Zalkonian undergoing transformation.

In this case, the cause for the failures is clear and the solution straightforward. With a single click on Retry all, ServicePulse sends those messages back to the original endpoint’s input queue. With things back to normal, those same messages can now be replayed successfully and your own order is on its way.

🔗NServiceBus vs. the Collective

In any world, message failure is inevitable, but NServiceBus and ServicePulse give your distributed system the tools to bounce back. With smart retry logic, UI-driven error recovery, and grouped failure analysis, you don’t have to wonder if your failed messages got assimilated by the Borg. In other words, resistance is not futile.

🔗Looking to improve failure recovery?

• Try recovering from failure using ServicePulse in one of our tutorials
• Learn about NServiceBus recoverability and retries
• Read the related blog post I caught an exception. Now what?

Let’s set the stage: It’s Return of the Jedi and the second Death Star looms large over the forest moon of Endor. The Rebel Alliance’s plan hinges on a synchronized attack — Han Solo leads the ground team to destroy the shield generator, while Lando Calrissian leads the space fleet to attack the Death Star itself.

For the mission to succeed, both parties must execute their part of the plan. If either side decides to abort, the other must as well, or it will be a disaster.

And here’s the twist: Han and Lando can only communicate through spotty, insecure rebel comms. Sound familiar?

🔗The Rebel messaging system

Let’s imagine Han and Lando are nodes in a distributed system. They need to coordinate a commit to the plan:

• Han says, “I’ll disable the shield.”
• Lando says, “I’ll attack once the shield is down.”

But Lando can’t attack until he knows for sure that Han will take out the shield. Han doesn’t want to risk the mission unless he knows Lando is ready to strike at just the right moment.

So Han sends a message:

“I’m ready to blow up the shield generator at 0300. Are you ready?”

Lando replies:

“Acknowledged. I’ll attack at 0300.”

But… what if Han never receives Lando’s reply?

Maybe the Empire is jamming the signal — those Stormtroopers aren’t known for their aim, but their comms interference is top-tier. Han now faces a dilemma:

• Proceed, risking that Lando never got the message.
• Wait, risking that Lando attacks without backup — or worse, aborts the mission.

Now Han tries to send another message:

“Got your confirmation—just confirming again we’re still go at 0300?”

And Lando has the same problem:

“Did he get my reply? Did he get my confirmation of his confirmation?”

Welcome to the infinite confirmation loop of the Two Generals’ Problem.

🔗No reliable victory without a reliable channel

The Two Generals’ Problem highlights a core truth in distributed systems: coordination over unreliable communication is fundamentally flawed. Even if messages arrive most of the time, we can’t be certain without an acknowledgment. And even then, we can’t be sure that the acknowledgment itself arrived.

Back in the day, we tried to solve this with distributed transactions, where technologies that use a two-phase commit algorithm like the Distributed Transaction Coordinator (DTC) would attempt to coordinate between databases and message queues (say, SQL Server and MSMQ) to ensure both the data and the message were committed atomically. The idea was noble: all or nothing across systems.

In practice, though? Depending on DTC was like relying on a Stormtrooper to hit a target directly in front of their helmet. Thankfully, we’ve moved past distributed transactions in modern architectures. But that doesn’t mean we can ignore the underlying problem. If anything, it means we have to solve it more thoughtfully.

Because the Two Generals’ Problem is hard, in distributed systems, we don’t try to solve the unsolvable. Instead, we change the game.

Avoid requiring perfect coordination. Don’t make success depend on both sides committing perfectly and deal with any mistakes that arise because of it. You can decide if this sounds like a plan and how to deal with it.

Design for uncertainty. Han’s team is intercepted, and when Lando goes to assault the Death Star, he realizes that the shield is still up. But importantly, neither side abandons the mission. They both keep retrying until they succeed.

We won’t get another chance at this, Admiral. Han will have that shield down. We’ve got to give him more time!

Use reliable message delivery. NServiceBus includes safeguards to ensure your messages don’t vanish into hyperspace.

Leverage the Outbox pattern. The Rebel Alliance makes the plan while everyone’s in the same room—synchronously agreeing on a coordinated two-pronged attack. In NServiceBus, the Outbox pattern ensures that either both missions are executed and succeed together, or neither is. Once the operation is in motion, there’s no turning back. Even when comms are jammed, and blaster fire erupts, both Han and Lando stick to the plan. The two generals are deployed in sync—after that, reliable communication is no longer assumed, but consistency is guaranteed.

🔗Trust the Force (or better: the Outbox)

In the real world, distributed systems don’t involve Death Stars or space smugglers (unfortunately). However, they do involve services trying to coordinate actions in the face of unreliable networks.

You can’t rely on perfect communication. But you can design systems that don’t break when communication is imperfect.

So the next time you’re designing a distributed system and thinking, “How can I make sure both sides agree before acting?” remember: Lando aborted and tried on a feeling. Don’t bet your business on a feeling, but use the reliability in NServiceBus.

If you feel like talking to one of our experienced distributed systems Jedi might help, transmit a distress call on the HoloNet. We’ll help you come up with a plan that ensures no Bothans will come to any harm.

Design for failure. Use the Outbox. Never, ever bet the galaxy on a single ACK. And may the 4th be with you.

You probably already know that, but I’d bet not all of your controller code is as lean as you’d like it to be. Is it?

So that leaves the question… How do we get there?

🔗Getting lean

What if I told you that you could make your controllers “lean and mean” and modernize in incremental steps?
Decoupling the web tier from processing logic is achievable through messaging. Sending messages from the web tier to dedicated backend message handlers allows for a gradual migration of complexity away from the controllers, preventing the need for a risky, all-at-once approach.
And as a bonus, your system gets reliability, resilience, and better scalability.

As you begin to move the complex logic out of the controller and replace it with sending messages and publishing events, your controller gets leaner, more manageable, and really quite boring.¹

However, there’s a catch. We want to maintain atomicity and consistency between our data and message operations. We don’t want data committed to the database unless the related messages/events get dispatched successfully. The same is true in the other direction. We don’t want any messages/events emitted unless the database transaction also succeeds.

Systems tend to use the transactional outbox pattern to ensure that data and messaging operations are kept consistent, and NServiceBus implements this pattern in its outbox feature, but that only works inside a message handler.

So, what do we do if we can’t avoid data operations in our controller actions and also need to send messages? Sometimes, this is unavoidable due to the design of a system. Some data has to be added to the database immediately, or the table view ² won’t be updated when the page refreshes. How do we maintain consistency between data and message operations then?

🔗Web tier consistency

The NServiceBus TransactionalSession feature allows the Outbox feature to work in the web tier. Using the TransactionalSession feature, you can store data and send messages in the web tier without having to refactor your controller or redesign your UI.

Combining the outbox with the transactional session will solve the problem of messages sent or published outside the context of a message handler while maintaining consistency between your message and data operations.

Let’s see how easy it is to set up this feature using NServiceBus.

The first step is to add a reference to the NuGet package related to your chosen persistence and register it. ³ Next, the NServiceBus configuration code will need to enable the Outbox and TransactionalSession features:

endpointConfiguration.EnableOutbox();//Each persistence has a specific Configure methodvar persistence = config.UsePersistence<YourPersistenceOfChoice>();persistence.EnableTransactionalSession();

Once you’ve configured the endpoint, you will have access to a session that can be obtained inside your controllers, and you can use them inside your controller actions.

public async Post(MyModel model, [FromServices]ITransactionalSession session){   await session.Open(new YourPersistenceOpenSessionOptions(),   cancellationToken: cancellationToken);   await session.Send(new YourMessage(), cancellationToken);   await session.Commit(cancellationToken);}

Now that everything is configured, you can easily benefit from using TransactionalSession in your controllers and leave the big refactoring for later.

🔗The big win

Previously, trying to introduce messaging to our controllers involved moving all of the logic to the back end to avoid consistency problems like zombie records and ghost messages. This wasn’t always possible without significant code refactoring or UI redesigns.

By using the TransactionalSession feature, you can still keep some of those mixed concerns in your controllers, while staying safe from any data inconsistencies in your system.
It is still ideal to keep the Single Responsibility principle in mind and, as much as possible, try to extract any data operations from the controllers to be handled asynchronously as part of a message handler. But that work can come later, you don’t have to do it right now.

In short, TransactionalSession mitigates risky and rushed code changes while maintaining consistency. You can defer refactoring to a later point when you’re not facing endless requests from the business stakeholders while racing towards hard deadlines and make these changes at a time when you have a bit more breathing room to refactor without burning the house down.

You can focus on your business priorities without investing time in a big refactor or overcomplicated configuration. The clean code aspect can be handled later.

Check out the transactional session documentation or download one of our transactional session samples to get started.

Using the transactional session keeps things simple and reliable; it guarantees atomicity with the infrastructure and technology already at your disposal. You get that rock-solid atomicity without the hassle of a massive overhaul or confusing setup. Your existing tools are all you need!

This post is the story of the growing pains experienced by our friends at VECOZO, a system integrator that ensures safe communication between numerous healthcare-related companies. They knew it would be irresponsible to design every piece of software to handle massive scale, even if it only had a few users. So, when the architects started to see telltale signs, they knew it was time to deploy more robust architectural patterns.

As you read on, maybe you’ll find that some of the challenges they faced sound familiar…

🔗Unblocking

Initially, an application could call another application using synchronous remote calls. During each request, the calling application had to wait for the remote system to complete. But that means when there’s a problem with one application, this affects other applications. For this, VECOZO came up with different solutions that suited their needs. They switched to asynchronous messaging instead of synchronous calls. But instead of manually creating a solution, they decided to use a ready-made framework called NServiceBus.

NServiceBus worked so well that they introduced messaging in multiple other applications and added more NServiceBus features to the system. Other departments caught on and introduced NServiceBus in their relationship management and healthcare purchasing systems.

Let’s look deeper at the problems VECOZO experienced with synchronous remote calls and what we need to remember when using messaging.

🔗When it’s synchronous

Synchronous remote calls, whether over HTTP or any other protocol, assume all involved parties are available, ready, and quick to respond upon request. A glitch in the request chain will ripple back to the original caller like a snowball rolling down a slope. If the caller is a few hops away from the failure, it cannot survive the error because it lacks context about the original reasons for the exception.

Systems evolve, and decisions made early can have long-lasting and unforeseen consequences. Usually, it’s not a problem until suddenly it’s a big problem. An experienced team doesn’t necessarily prevent every single issue like this, but when they happen, they diagnose it quickly and take proper steps to mitigate it, as was the case here.

When VECOZO started suffering from those effects, they laid out a plan to address the limitations of the current design, namely, how to reduce the temporal coupling introduced by the synchronous remote procedure calls.

🔗What about retries

To solve the issue, the team could have introduced a short-term fix to include automatic error retry mechanism as a short-term solution. When a remote procedure call failed, the calling code would retry it.

However, determining an appropriate retry strategy can be more art than science. How many times do you retry? Do you use an exponential backoff strategy? What happens if your retries inadvertently cause a denial of service attack? After investigating this option, the team realized such a solution might not be ideal.

🔗Messaging, a better approach

A message-based solution replaces synchronous procedure calls with sending a message asynchronously. Essentially, instead of saying, “Hey claims service, can you perform layout checks on this claim? I’ll wait,” the calling process would say: “Hey claims service, can you perform layout checks on this claim? Take your time; I’ll continue when you’re done.”

Now, whether a component was available or not didn’t matter anymore. Async messages replaced direct calls. Messages are naturally stacked up in a queue, waiting for the component to become available again. So, while some of the team still working on a home-grown solution was looking into rate limiting, the developers on the team switching to NServiceBus worried much less about overloading their components. They experimented with the optimal number of threads that would process messages in parallel. As a result, the components could go as fast as possible without ever overloading resources like a database.

That’s not to say that the team ignored a retry strategy. Since NServiceBus has this functionality built-in, it was easy to enable it. But as a bonus, when a process did fail, the offending message could be delivered to an error queue along with the context of the failure in the form of the exception details. The operations and development teams could work together to investigate the issue, define a fix, deploy it, and finally put the message back in the queue to retry. The business processes could resume where they left, with only a short delay.

🔗Messaging considerations

Things are never as straightforward as they appear on the surface, and moving from synchronous calls to asynchronous ones is no different. Due to their nature, synchronous calls often rely on ordering. Things will happen as they are declared in code—one after the other. Changing processes to be asynchronous introduces entropy. Processes can no longer rely on the steps’ execution order, as messages are processed out of order, and thus, certain components require some redesign.

Another key difference between retrying a synchronous remote call and retrying a message is who’s responsible for what. When the invoker sees its call fail and needs to retry, it has little to no knowledge about the context of the failure, and as such, its options are limited to a backoff retry policy.

When using messages, the failing message fails at the receiver end. The receiver has all the context and knowledge to make more educated decisions about how to retry and whether it’s worth it.

By forcing the sender/invoker to try, we’re violating an ownership boundary by making the receiver problems a sender/invoker concern. We should never offload issues to someone with little to no knowledge of addressing them.

🔗Retrospective

Even though NServiceBus provides developers with the flexibility to do what they want in code, the team appreciated the NServiceBus “pit of success” philosophy, which makes it harder to do things the wrong way. Best practices are everywhere, embedded inside NServiceBus and its API and in great documentation. It provides a standard way of working, percolating throughout the system. The team especially appreciated the Particular Software support team, consisting solely of NServiceBus developers with experience building complex, distributed systems. Having one solution removed the need to create, maintain, and document a homemade framework.

The focus on code was one of the best features that made the team choose NServiceBus. The developers already felt most comfortable in a code editor, with the ability to safely commit changes to source control.

Automatic retries, error queues, and the design of loosely coupled event-driven applications paved the way for adding new functionalities by adding new subscribers for existing events, as well as no-downtime releases during working hours. Developers and operations personnel had the safety measures to release confidently.

🔗Summary

Any software system used for a considerable amount of time will go through growing pains as it graduates from prototype to minimum viable product to critical business system. The patterns used to accelerate its development in the early stages can’t always provide the stability and scalability required later in life.

The key for software professionals is to recognize the patterns that indicate a system is beginning to grow beyond its architectural capabilities and how to replace earlier architectural patterns like HTTP and synchronous calls with more robust patterns like asynchronous messaging.

Curiously, what’s often missing from the discussion is details, like how to handle all the other on-prem systems that integrate with your system, both upstream and downstream, that can’t be upgraded at the same time. This gets even more tricky when those integrations are over on-prem-only technologies, like MSMQ, that don’t integrate out-of-the-box with cloud alternatives like Azure Service Bus or Amazon SQS. It’s as if they’re saying, “Have you documented your system? Great! Have you chosen a cloud provider? Awesome! Do you have all the services in place? Wonderful! Now go rewrite all your code… we’ll wait…are you done yet?..What are you looking at me for? I’ve already told you to plan carefully, I can’t do EVERYTHING for you”

In short, there’s a big gap between “everything works on-prem” and “everything works entirely on the cloud” that often gets glossed over. So we’re going to explore this scenario with a small, fictitious airline, called (and really, what else would we call it) ParticulAir.

🔗I want to move one of my on-prem systems to the cloud

ParticulAir has a legacy system that’s been running successfully for many years with a number of features, including flight upgrades. These upgrades are handled asynchronously, as the airline wants to prioritize upgrades for its most valuable frequent flyers over others. Technically, this is all done over MSMQ where requests are processed and eventually granted or rejected, notifying other services of the outcome. Here’s a simplified diagram of how that works:

Now, the business wants a new mobile app that will enable users to do all of the things currently available over the web, including requesting flight upgrades. They’re also thinking of migrating the legacy system to the cloud to save on costs, get dynamic scaling, and all the other benefits of the cloud.

While they would like to eventually migrate/refactor/rewrite the system to be cloud-native, that could take potentially years for a big system. However, if they could get that new mobile app up and running by integrating it with the existing systems, that shorter time-to-market would definitely be appreciated.

Luckily, there’s a way to do just that.

🔗The Messaging Bridge Pattern

The Messaging Bridge is an intermediary component that receives messages from one queuing system and transfers them to a compatible queuing system elsewhere.

In ParticulAir’s case, that would mean the cloud-hosted back-end for the mobile app would put a message in the relevant cloud queuing service (Azure Service Bus or Amazon SQS) and use the “bridge” to route it to the legacy MSMQ on-prem system. Here’s what that would look like:

The immediate benefit of a bridge in this scenario is that new functionality (e.g. the mobile app) can be built using modern, cloud-based technology while still leveraging the tried-and-true code in the various legacy systems. This provides some breathing room for the cloud migration. New features can be added without having to re-write the legacy system at the same time. Even better, depending on the implementation of the bridge, the legacy systems may not even need to be touched at all. As long as they receive the MSMQ messages with the required data, they shouldn’t care where it originated.

Now eventually, ParticulAir does want to migrate their systems away from the on-prem, MSMQ technology. This is another instance where the Messaging Bridge Pattern can help. With a bridge in place, the entire system doesn’t need to be migrated all at once. Instead, a more gradual process can be used, moving one endpoint at a time from MSMQ to the cloud, with the bridge transparently taking care of the routing. This can remove a lot of the complexity and risk inherent in a large-scale migration. Let’s see how with an example.

🔗Don’t everyone migrate all at once now

Remember from the diagram above that the Upgrade component also publishes UpgradeFulfilled events that the Marketing component listens to (all using MSMQ). When that Upgrade component is migrated to Azure, then when it publishes those very same events, they will go to an Azure Service Bus topic called “UpgradeFulfilled”. With a bridge in place configured to route messages from the “UpgradeFulfilled” Azure Service Bus topic to the MSMQ “UpgradeFulfilled” queue, the Marketing component can continue running unchanged in the on-prem environment.

Without using some kind of bridge, both components would need to be migrated or at the very least “duplicated” (after modifying the on-prem component to talk to a cloud-accessible database). The thing is, that Marketing component probably talks to other components itself, which then would have to go through the same migration or duplication exercise (together with the components they talk to).

This is far riskier than if just one component could be migrated, because it means all those components would need to be tested and deployed in tandem. Imagine if any issues arose during testing or (shudder) in production, and you had to pinpoint where the problem lay. This is much easier if you deployed only a single component compared to a series of interdependent ones. Not to mention that it would be far easier to roll back a single component to a previous version. It gets even more complicated if the different components are managed by different teams.

All of this would also slow down the timeline for the mobile application to release its’ flight upgrade feature.

These problems go away if you have the ability to migrate a single endpoint at a time. Once a messaging bridge is in place and configured, teams can migrate their endpoints however they see fit without worrying about how their outgoing messages get routed to other endpoints.

So far, we’ve been almost as hand-wavy as most of the traditional cloud migration literature has been. It’s all well and good to say, “just use a bridge” but how do you implement one?

Here’s the good part: you don’t have to.

🔗The NServiceBus Messaging Bridge

The NServiceBus Messaging Bridge was designed specifically for these scenarios. It’s an implementation of the Messaging Bridge Pattern that takes care of routing messages between different queuing systems.

In our initial mobile app Flight Upgrade scenario, the bridge sits between the Azure-hosted mobile back-end, routing messages from Azure Service Bus to the MSMQ-based on-prem system:

The code in the mobile back-end can send messages to an Azure Service Bus queue named the same as the MSMQ queue of the on-premises upgrade component, say, “ParticulAir.UpgradeService” ignoring that a bridge is being used at all, as if the upgrade component was also hosted in Azure. By configuring the NServiceBus Messaging Bridge appropriately, messages from that Azure Service Bus queue will be forwarded on to where they need to go transparently.

This means when we eventually migrate our Upgrade component to Azure, listening to the “ParticulAir.UpgradeService” Azure Service Bus queue, we won’t need to touch our mobile back-end. Instead we would re-configure the Bridge to stop listening to the “ParticulAir.UpgradeService” queue and have it listen to the “ParticulAir.UpgradeFulfilled” Azure Service Bus topic, forwarding those events over MSMQ to the downstream Marketing component, which wouldn’t need to be modified either.

Through this process, we could migrate all the relevant components in this scenario, one at a time, to run on the cloud. When the last component is migrated, we’d remove the Bridge from the solution completely.

Until this goal is met, however, the messaging bridge can make sure your migration can happen safely and in smaller, more manageable chunks.

🔗Summary

We’ll reiterate the standard advice that migrating a complex distributed system to the cloud requires a well-planned, incremental approach that maintains system integrity and minimizes risks.

The Messaging Bridge Pattern can be a crucial component to your migration and, if your system uses NServiceBus, you can even wash your hands of most of the implementation details.

To see it in action, check out our sample on bridging messages between endpoints using MSMQ and Azure Service Bus.

When one of those meetings occurs after a carb-heavy lunch, it’s easy for your mind to drift away…back to those university lectures about entity design. Think of the nouns and what attributes they have. A dog and a cat are both animals and have 4 legs. Except now it’s Customers, Orders, Products, and Shopping Carts.

Is this the best way to build a system, though? Didn’t we do the exact same thing on the previous greenfield project we’re now rewriting? Surely we won’t make all the same mistakes we made last time…right?

🔗One cart to rule them all

As the meeting continues, the design for the shopping cart begins to take shape. ShoppingCart is a noun, after all, and it’s got a list of items in it, each of which has simple attributes like Price and Quantity. Here’s the shopping cart part of the entity relationship diagram we’ll print out and keep at our desk ² like a holy article of software design scripture:

We’ve also realized that a cart has some behavior associated with it as well, operations like AddToCart(), SaveForLater(), and Checkout(). So we’re now combining data and behavior together…this is essentially an aggregate which means now we’re doing domain-driven design!

🔗More attributes, more problems

During development, we start to see some flaws in the plan.

First, we learn that if the price of an item goes down, the new lower price should also be reflected in the shopping cart. So whenever a price changes, we must copy that value to any shopping cart containing that item. However, if the price of an item goes up, we need to warn the user about it and make them accept the new price. So now the cart items need to store the current price and the previous price, and we have to do a lot of copying whenever any related data changes.

Next, we realize that we need the inventory level to accurately reflect the available inventory in the warehouse. The business intends to use this to pressure customers to purchase before it’s gone. ³

To keep this value up-to-date, every time the inventory of any item changes in the warehouse, we would need to check every active shopping cart for an instance of that item and update its value. You may be able to join tables to get this information, but that’s not always an option. For example, you might need the data to be denormalized for performance, or the warehouse data might exist on a physically different system that can’t participate in a database join.

It gets worse. As it turns out, we discover similar concerns around delivery estimates, item names, and descriptions. So every time any of these values change, they’ll also need to be copied from their source of truth to any shopping cart with a matching item. At least the marketing folks insist that changes to product names and descriptions should be infrequent and primarily limited to typos. Let’s hope that’s true.

So now our shopping cart starts to look a lot messier, and we’re starting to get worried thinking about all the batch jobs we’ll need to write to keep this thing updated.

Our Cart object no longer looks like a proper DDD aggregate, with everything dependent upon everything else and data being copied everywhere.

The sinking feeling of déjà vu from the old project starts to creep in. What happened? And, more importantly, how can we fix it?

🔗Anti-requirements to the rescue

To help decompose a complex domain, we can use anti-requirements ⁴ to find attributes incorrectly lumped together on the same entity. Using anti-requirements is a powerful way to increase autonomy by breaking your domain into separate islands that can evolve independently. ⁵

Anti-requirements are deceptively simple: you create some fake requirement concerning two attributes and present it to business stakeholders. “If the product has more than 20 characters in its name,” you say to them, “then its price must be at least $20.”

When they laugh at you, that’s a hint that although those two attributes are verbally associated with the same noun, there isn’t any meaningful logical relationship between them. ⁶

Without anti-requirements, teasing out these details can be tricky. Since business domain experts tend to think of this stuff as obvious, which makes them unlikely to volunteer this information. They’re generally surprised that developers don’t know it already. That makes it our job as developers and architects to dig for it.

So with this in mind, let’s go back to our shopping cart and ask ourselves: Will the business people think I’ve lost it if I ask what business rules might operate on Attribute A and Attribute B? If the answer is yes, you’ve likely found an anti-requirement.

🔗A new and improved cart

Let’s start teasing out some anti-requirements and see what effect that has on our shopping cart, beginning with the concept of price.

• When the price of a product exceeds $100, the name should be changed to all caps. Ridiculous!
• When a product description is longer than 3000 characters, the price should be increased by 10%. Ludicrous!
• When the inventory for an item is higher than 1000, we should charge 10% more. Inconceivable!

But wait, we need to be careful. When hearing that last anti-requirement, our business stakeholder might say that while that is indeed inconceivable, it could be possible that we’d need to charge more when inventory is low. After all, that’s just supply and demand in action. By using anti-requirements in this way, you might accidentally discover business requirements that could have been overlooked otherwise.

But whatever anti-requirements we dream up, it remains clear that price and quantity are related. After all, you must multiply price × quantity to get the total cost.

This suggests that the highly-coupled price and quantity values could be extracted elsewhere.

In the same way, we can start to analyze other pairs of attributes, crafting anti-requirements for each and using how ridiculous they sound to determine whether to extract other groups of attributes that are more tightly coupled.

• The name of a product affects estimated delivery because we ship products alphabetically. Absurd!
• We must update the description of an item every time the inventory level changes. Preposterous!
• The more inventory we have of an item, the longer it will take to ship them. Wackadoodle! ^{7 8}

Remember that shopping cart entity? We used anti-requirements as a club to bash it into pieces. It turns out that while a shopping cart is a noun used by the business, there is no “cart” anymore…only a simple CartId rather than a full-blown entity or aggregate.

Eagle-eyed readers will notice here that the Quantity is not owned by any one thing but is shared between Sales, Shipping, and Warehouse. It’s important to realize that even single attributes don’t always mean the same thing. In Sales, quantity is a multiplier for the price. In Shipping, it’s how many items to put in a box…or even multiple boxes. In Warehouse, it’s how many things to reserve and restock. The values just happen to come from the same place, and we’ll show how to handle that a little later.

This shows that not all the nouns the business uses need to have a corresponding entity in your domain model.

🔗Improved efficiency

Only grouping together data that changes together has a lot of technical and organizational advantages as well.

From a technical perspective, attributes that change together should also be cached similarly. For example, after a product is published, its name and description do not change frequently and can be cached for a long time, but price and inventory would probably change a lot more. Storing those attributes in different entities allows us to use the most appropriate caching strategy for each. In our case, serving the product name and description from a JSON file hosted on a content delivery network (CDN) might be a better and more scalable approach than using a server-side cache like Redis.

In fact, if you’re storing product images on a CDN based on a convention like https://mycdn.com/products/{ProductID}/{Size}.png, then you’ve already begun decomposing your domain using these strategies.

From an organizational perspective, you no longer need to get all the business stakeholders together simultaneously. ⁹ If you need to add the capability to deliver digital goods that don’t require shipping, the number of people that need to be involved is now significantly reduced to only the people with insights into the relevant parts of the “cart.”

The only remaining problem for our shopping cart is taking all of Humpty Dumpty’s pieces and putting them back together again.

🔗ViewModel composition

Everything has a cost, and decomposing using anti-requirements is no different. Each component gains greater autonomy, but it can feel (at first) that this comes with a price of greater complexity.

Our users still think of a “shopping cart” as a thing and expect to see all the attributes we’ve separated on a shopping cart page together.

We can integrate all the shopping cart attributes on the same page using a strategy called ViewModel composition using tools like ServiceComposer where independent components provided by services or microservices ¹⁰ can query their own data from disparate back-end systems and combine it into a single ViewModel without reintroducing coupling at the UI layer.

In ViewModel composition, each component registers its interest in providing data for specific URI route patterns. Then, for each web request, all the interested data providers are asked to fetch their data, which is added to a dynamic ViewModel. Finally, a separate service (let’s call it Branding) takes the ViewModel and renders it to HTML.

The story is similar for POST requests. Components register handlers for POST routes to communicate back to their respective back-end systems, usually with async messages. This is how each service persists the Quantity value back to its own data store without any knowledge of the other components.

Using ViewModel composition does add some complexity, at least in the short term. It will not make building the first screen faster, but it will make building the 10th, 50th, and 100th screen faster. Limiting unnecessary coupling, even in the UI layer, makes it easier to continue creating new features well into the future.

ViewModel composition techniques also allow flexibility in terms of data storage technology. For example, one service could be powered by a traditional relational database. At the same time, another could use a graph database or key-value store, a heavy caching layer with Redis, or even JSON files on a CDN.

And when a single service only has to worry about its own vertical slice ¹¹ of the overall system, you’ll find that a database diagram actually can fit comfortably on a single sheet of paper.

There’s much more to say about ViewModel composition than can be covered in a single blog post. Check out Mauro Servienti’s blog series on ViewModel composition or his webinar All our aggregates are wrong ¹² to learn more.

🔗Summary

In today’s increasingly complex software systems, the “noun has an attribute” approach to modeling is bound to result in classes, components, and systems that become a mess of coupling. Anti-requirements are one strategy we can use to find our logical service boundaries, helping us to discover which attributes belong together and which have no business being anywhere near each other.

Over time, too much coupling causes the system to evolve into a big ball of mud. Eventually, making changes anywhere without breaking something seemingly unrelated becomes impossible.

Organizations that decouple into autonomous services will be able to be more nimble and deliver value to the business years into the future. After all, unlike most other business projects, software isn’t ever really “done”.

Everyone else will be stuck rewriting the system in 3 years. Again.

However, in a message-based system, we no longer have a single call stack. We’ve exchanged it for a haystack of call stacks, which makes finding the needle (the root cause of the failure) even more difficult.

🔗The problem

Because message-driven systems are asynchronous and run in multiple processes, debugging is naturally more complex than in a single-process application. Failures surface as failed messages in a specific endpoint, which could be symptoms of an issue happening further upstream. We need to understand how and where the business transaction started, how the message flow works, the order in which messages were processed, and have a thorough understanding of the components involved.

Debugging these issues is tedious, cumbersome, and sometimes downright painful. In addition, setting up a debuggable environment is a serious challenge as we need to understand the components involved in the scenario we’re debugging, which could span different solutions, repositories, or endpoints, each of which may require additional infrastructure to be available.

Luckily, we can address these concerns with observability.

🔗Observability

Observability is how well we can figure out what we don’t know when it comes to system behavior based on its external outputs. For example, in a highly observable system, we can easily infer the internal state and behavior of the system without going back into the code to make changes when something unexpected occurs.

The Particular Service Platform includes a wide range of observability features, including ServiceInsight’s sequence diagram and flow diagram. When auditing is enabled, ServiceInsight can visualize the flow of messages across multiple NServiceBus endpoints. In addition, you can inspect the message headers and body, allowing us to figure out what’s going on in any flow of messages.

ServicePulse also exposes a range of metrics that provide insight into the system’s behavior. One of the main benefits of the Particular Service Platform is that it allows for black-box instrumentation. Apart from enabling auditing and metrics in your configuration, it doesn’t require any code changes to gather instrumentation from your endpoints.

However, distributed systems are inherently complex and consist of many components: message brokers, databases, distributed caches, integration points, REST APIs, front-ends, and more. All of these components can participate in a business transaction. Still, the information captured in messages may represent only a small part of that business transaction—not enough to tell the whole story.

🔗A sample interaction

Let’s consider a system with an order process flow that looks like this:

flowchart TD    A[ASP.NET Core OrderController] -->|1. Place order| B[Sales]    B --> |2. Start order process| C[Order saga]    C --> |3. Charge order| D[Payments]    D --> |4. Order charged| C[Order saga]    C --> |5a. Ship order| E[Shipping]    C --> |5b. Bill order| F[Billing]

In the controller method, nothing fancy is happening; we’re just mapping the data collected in the web request to a message and sending it to the Sales endpoint.

But what if we mess up and swap the shipping and billing addresses?

As far as the Sales endpoint is concerned, that input is perfectly valid. But unfortunately, the endpoints cannot validate that we didn’t swap the data, and inspecting the message bodies won’t give us insight into where we messed up. After all, both look like valid addresses. So now we’re shipping orders to the wrong address and wonder why so many customers are complaining they never received their order.

We’re facing a blind spot because we can’t investigate what context, decision-making, or flow led to the creation of that initial message. We can’t see the entire story of the interaction all at once. The example might be simplistic, but it shows why we need insight into the whole business transaction, especially in large and complex distributed systems where additional decisions are made every step of the way.

🔗OpenTelemetry and NServiceBus

Enter OpenTelemetry: a vendor-agnostic, cross-platform, and open-source standard for observability to help standardize how we instrument, collect, and export instrumentation from our applications. OpenTelemetry includes tools and SDKs for each programming language to capture and export your applications’ metrics, traces, and logs.

And now, OpenTelemetry is available in NServiceBus.

OpenTelemetry tracing was adopted in .NET in the System.Diagnostics.Activity API in .NET 5. ¹ It’s also available for earlier versions ² using the dedicated System.Diagnostics.DiagnosticSource NuGet package.

With the advancements made in the industry for observability with OpenTelemetry, we want to strengthen the observability of the platform further. Therefore, from NServiceBus version 8 onwards, you can enable OpenTelemetry on your NServiceBus endpoints to seamlessly capture instrumentation and export it to your observability backend.

By enabling available instrumentation libraries, we can observe quite a bit without adding application-specific tracing. For example, we can solve the address-swapping example above by enabling instrumentation in both ASP.​NET Core and NServiceBus, and then adding information to the spans emitted by these libraries. The ASP.​NET Core instrumentation library creates a span that represents the request submitted by the user, to which we can add a span containing the shipping and business address. NServiceBus already adds a span when ingesting a message from the queue, at which point we could add any other relevant tags.

Whether in a controller method or a message handler, we could add code like this:

Activity.Current?.AddTag("order.id", request.OrderId);Activity.Current?.AddTag("order.shipping_address", request.ShippingAddress);Activity.Current?.AddTag("order.billing_address", request.BillingAddress);

Based on the added tags, if we carefully compare the initial request’s tags with the message processed in the endpoint, we would quickly identify that we reversed the billing and shipping addresses.

Note: Even though we’re using addresses in this example, adding personally identifiable information (PII) to telemetry is never a good idea! Remember that this information is sent to an external system, your observability backend. Imagine getting a right to be forgotten request to remove data when you have exposed personal information in your telemetry. The flow in this article is just a simple example to illustrate the point…and an opportunity to remind you of this best practice at the same time.

Here’s a screenshot of how an enriched span would look like in Jaeger:

…but we could also use Azure Application Insights, Elastic, Honeycomb, or a different tracing backend. The main advantage of using OpenTelemetry is that you can use any vendor that best suits your needs without changing the instrumentation code or libraries. The only change required would be configuring the appropriate exporter to send your instrumentation data to the right place.

With OpenTelemetry enabled across all components of a system, we can create system-wide observability and understand which components take part in the business transaction from beginning to end.

To enable OpenTelemetry in your NServiceBus endpoint, add the following:

endpointconfiguration.EnableOpenTelemetry();

NServiceBus will then capture instrumentation for every incoming and outgoing message operation in an endpoint. If the underlying message queue initiates a trace, NServiceBus will create a child span when processing that message. Otherwise, NServiceBus will create a new one. Any subsequent outgoing operations result in child spans on the incoming message span.

OpenTelemetry operates under an opt-in model, so even though OpenTelemetry is enabled on the endpoint, you still need to set up a TracerProvider to collect that instrumentation and export it to a tool of your choice.

var tracingProviderBuilder = Sdk.CreateTracerProviderBuilder()    .AddSource("NServiceBus.Core")    // ... Add other trace sources    // ... Add exporters    .Build();

Then you can see the full trace:

If you want to enrich the telemetry created by NServiceBus, you can do so inside the message handler. You can either start a dedicated span, which will automatically be created as a child span by the .NET Activity API and propagate the context as expected, or add additional tags and events to the active span without creating a new span, as in the example. If you want to add specific details across all spans, you can customize the OpenTelemetry traces using a custom processor.

Logging is important too. We’ve been doing that for years, but with OpenTelemetry’s focus on telemetry correlation it gets a lot more powerful. We can connect those logs with the traces we’re now collecting to tie it all together. Check out our sample on connecting OpenTelemetry traces and logs to see how to do that.

🔗Summary

Ready to try it out on your own? In that case, we have multiple samples available for you to try out, including dedicated code samples for Prometheus and Grafana, Azure Application Insights, and Jaeger. As you use OpenTelemetry to improve your root-cause analysis, please let us know how it’s going and how we can further improve the observability of the platform.

All this is baked right into NServiceBus version 8. However, if you’re still using NServiceBus version 7, you can use Jimmy Bogard’s community package ³ to bridge the gap.

When you’re debugging a distributed system, haystacks are everywhere. But with OpenTelemetry, you have the necessary tools to find the needle.

I recently had the opportunity to chat with the lead developer for the luggage arrival system at a major Asian international hub. ¹ He told me how using NServiceBus made it possible to get all the systems in the airport to work together reliably.

🔗Reading from the firehose

When an aircraft is inbound to the airport, the first step is to read all the bag source messages representing the bags in the aircraft’s hold ² from a service called ARINC BagLink.

ARINC is a global service used by many airlines, but it’s not the easiest service to work with. Once you connect and subscribe, you process the information coming in through the TCP connection. This is not JSON or XML information—it’s a low-level byte stream with fields that are fixed numbers of bytes. Only an arcane set of rules determines what constitutes a message, let alone how they’re formed. Translating the stream of incoming bytes into a series of messages can be tricky, given that there’s little to distinguish even where one message ends and the next begins.

The biggest challenge here is how to reliably process this data because it’s not straightforward to reread the information from ARINC. If you miss it the first time, you have a problem. Usually, messages only arrive at a rate of about 20 messages per second, but the system needs to be reliable and scalable enough to handle between 1000 to 2000 messages per second.

The only safe way to deal with information like this is to immediately append it to a file on disk, and then pass it to a message queue using NServiceBus.

Once the bag source message information is contained in an NServiceBus message, we don’t have to worry about losing it. The recovery capability in NServiceBus makes sure that any failure will go through a series of message retries in case the error is transient or is safely written to an error queue where developers can inspect the problem, retry the messages, or even fix malformed messages before retrying them.

Without processing the bag source messages through a queue, it would be impossible to complete all the steps required for the processing of each bag message—including parsing the messages, saving them to a database, and then matching up bag information with flight information—at the speed required by the incoming data stream coming from ARINC. Additionally, any faulty data or flaw in application logic could result in a loss of baggage information from ARINC that can’t be easily recovered.

🔗Flight tracking and matching

Incoming luggage has to be matched up with flights as well. Flight information comes from a separate flight information system, which is the same system that drives the large Arrivals and Departures screens inside the airport terminal.

Then, baggage and flight information must be joined together into a bag list containing a bag number (found in the barcode attached to the bag), a flight number, and a departure date (typically with no year) for each bag.

The problem is that it’s actually really complex to uniquely identify a specific flight.

Assuming a made-up airline code XX, the canonical form of a flight number is XX0460, but some systems might represent that as a shorter XX460, but that’s just the start of it.

Depending on the flight, the arrival time could differ significantly from the departure date, especially for long-haul flights crossing the Pacific Ocean and the International Date Line. But there are other factors, such as if the flight gets delayed, or canceled and rescheduled. Even a canceled and rescheduled flight would carry the original departure date—not necessarily the date the bag got loaded onto the plane.

Flights can also be cross-listed on other airlines, such as when a Delta flight is “operated by” KLM, one of its airline partners.

A myriad of logic like this goes into matching bag numbers with the flights. Using NServiceBus allowed the lead developer to divide this logic up using different message handlers and publish/subscribe techniques so that they could design the overall flow of messages through the system, and pass off the implementation of individual message handlers (representing much more contained and well-defined problems) to other developers on their team.

🔗On arrival

When a flight arrives, some bags will be routed to connecting flights, while others will make their way to the arrivals hall to be picked up at a baggage carousel.

For the bags headed to connecting flights, NServiceBus message handlers translate queue messages back to the byte-level protocol to be transmitted back to the ARINC service. But for the bags headed to the arrival hall, the lead developer wanted to automate the system that displays which flights are being served by which carousels and the status for each carousel.

As the bags are unloaded, a baggage crew member armed with a barcode scanner scans the barcode on each bag before it is placed on the conveyor belt. The scanner connects to an API that generates an NServiceBus message, and then one by one, each bag on the bag list is accounted for.

When each bag is checked off the list, the bag arrival system is automatically updated to display that all bags have been unloaded. It’s at this moment that, unfortunately, some weary traveler might realize their bag isn’t going to appear on the carousel after all, and they will need to go report a lost bag.

Many of these systems can be overridden manually, for instance, to say that all bags have been unloaded. Still, for the most part, the automation that occurs by tracking each bag allows the whole system to operate completely autonomously, allowing operators to focus on other tasks.

🔗Summary

An airport is a prime example of a very public place where a whole lot of software and a litany of business rules come together in ways that most people never think about, let alone fully comprehend. By comparison, other business domains might appear simple at first, but every domain has this tendency to hide its own complexity until you really start to dive into the scenarios.

In all these business domains, NServiceBus can provide the ability to break down this complexity. Each problem gets broken down into processes, each process into a series of steps, and each step is represented as a message handler processing a message.

The safety of business data encoded into messages means you can safely read from a firehose of external data, knowing you can’t lose data. The discrete nature of messages means it’s easier to reason about what’s happening within one well-defined interaction or related to one specific business rule. And the ability to build orchestrations around the results of multiple messages makes it easier to design and build processes around individual events over longer periods of time.

Air travel is only one example. So how could NServiceBus make your business domain better? Give us a shout, and let’s talk about it.

In NServiceBus 8.0, which is now available, we have introduced support for cooperative cancellation, which will give you greater control over how an NServiceBus endpoint behaves when you need to shut it down.

Let’s talk about what cancellation is, how it relates to NServiceBus, and how we’re delivering cancellation in NServiceBus 8 without forcing a massive breaking change on your existing systems.

🔗What is cancellation?

As simply as possible, cooperative cancellation in .NET means passing around a CancellationToken that is associated with a CancellationTokenSource so that the code observing the cancellation token can later be told to stop or cancel.

You can observe a token and then return from a loop when the token is signaled:

public async Task BigLoop(CancellationToken cancellationToken){    while (!cancellationToken.IsCancellationRequested)    {        // Do more work    }}

However, if you consult Microsoft’s recommended patterns for CancellationToken, it’s better to always throw an OperationCanceledException so your caller knows that the work was interrupted:

public async Task DoSomething(CancellationToken cancellationToken){    while (KeepDoingStuff)    {        cancellationToken.ThrowIfCancellationRequested();        // Do more work    }}

When calling a cancellable API, you can create a CancellationToken that automatically expires after a given amount of time, or you can use CancellationToken.None or default to say you don’t care about cancellation and that you want the method to complete no matter what:

// Cancel after 10 secondsusing (var tokenSource = new CancellationTokenSource(TimeSpan.FromSeconds(10))){  var token = tokenSource.Token;  await DoSomething(token);}// Do not cancel (the following two lines are equivalent)await DoSomething(CancellationToken.None);await DoSomething(default);

If you’re designing an API, you can also make the cancellation token optional:

public async Task DoSomething(CancellationToken cancellationToken = default){    // Do stuff}// These calls are equivalent due to the parameter defaultawait DoSomething();await DoSomething(CancellationToken.None);await DoSomething(default);

There are a lot of other fancy things you can do with cooperative cancellation, but let’s talk about what cancellation means for NServiceBus.

🔗NServiceBus and cancellation

In an NServiceBus endpoint, the main reason to care about cancellation is when the endpoint is shutting down. When the endpoint shuts down, you want it to stop cleanly in a reasonable amount of time without having to forcibly kill the process. This is especially true if, for example, you are hosting your endpoints as Docker containers in Kubernetes, and the cluster needs to move your node from one host to another.

In this case, you want to stop receiving new messages from the queue, then give the existing “in-flight” messages a chance to complete processing successfully before shutting down the process cleanly.

But as a developer coding a message handler, there’s nothing you can do in NServiceBus version 7 because you can’t access a CancellationToken within that handler.

If a handler just started running a SQL query that could run for 100 seconds, what do you do?

In NServiceBus version 8, you can access a CancellationToken and pass it to the SQL query so you can shut down efficiently. We’ve also done it in such a way that you won’t have to change every single message handler.

🔗Breaking changes

Correctly implementing cancellation is an important addition for NServiceBus, but we couldn’t do it without breaking changes. That’s why we did it in a major version, which is NServiceBus 8.

The accepted method of implementing cancellation is to add CancellationToken parameters to all async methods and for a caller method to pass the token along to the callee, kind of like a bucket brigade passing the same token value all the way down the call stack.

This pattern is supported by tooling in Visual Studio. For example, there’s a Roslyn analyzer CA2016: Forward the CancellationToken parameter to methods that take one that reinforces this pattern and offers a code fix that will fix all violations of the rule, even in an entire solution, with just a few clicks.

Just click the Solution link at the bottom of the pop-up and Visual Studio will fix every instance where you forgot to forward the CancellationToken parameter. It’s pretty slick.

Unfortunately, CA2016 is only an informational message, not a warning or build error. And you might not even have the analyzer at all unless your project is set up correctly.

To make sure you forward cancellation tokens correctly, we recommend enabling the built-in .NET Analyzers and using an .editorconfig file to upgrade CA2016 to a warning or error to make it more visible:

[*.cs]# Make it a green squiggledotnet_diagnostic.CA2016.severity = warning# Or make it a red squiggle that fails the builddotnet_diagnostic.CA2016.severity = error

To implement cancellation properly, we have to add a CancellationToken parameter to every async method. For a class method, that’s not such a big deal — you just add CancellationToken cancellationToken = default to the parameter list. In most cases, someone calling that method can recompile with no changes required.

But with interfaces, it’s a different story. For example, here’s an interface before and after adding cancellation:

// No cancellationpublic interface IDoSomething{    Task DoIt();}// Cancellation addedpublic interface IDoSomething{    Task DoIt(CancellationToken cancellationToken = default);}

However, it doesn’t matter that you marked the parameter as optional. Because an interface is a contract, a class implementing that interface must add the token parameter:

This change to an interface is a breaking change, and worse, a change that is difficult to decorate with Obsolete attributes to guide the user in their upgrade. We start in a bad spot because the feedback from the compiler here is not helpful—instead of saying that you need to add the CancellationToken to the DoIt method, it instead highlights the IDoSomething interface name, saying the class doesn’t implement the interface anymore. To make matters worse, if you let the compiler “fix” it, it will create a new DoIt method overload that contains the token.

All this brings us NServiceBus’s central interface, IHandleMessages<T>.

🔗IHandleMessages<T>

Every message handler in an NServiceBus system is a class that implements the interface IHandleMessages<T>. Here’s the interface definition:

public interface IHandleMessages<T>{    Task Handle(T message, IMessageHandlerContext context);}

If we changed that interface, users would have to change every single message handler class in their system. That would be an excruciating change ¹ to force onto our users, and we’d prefer not to do that if we can help it.

So, on the one hand, we have the generally-accepted way of implementing cancellation, backed up by compiler support through Roslyn analyzers, demanding that we break the interface. But on the other hand, we have all the pain we would cause customers by releasing a breaking change that affects nearly every code file where NServiceBus is referenced.

How do we reconcile these two sides?

🔗So what’s changing?

Luckily, we found a way to support cancellation and keep the benefit of compiler support but not force you through the pain of an IHandleMessages<T> change.

On all of our lesser-used methods and all our internal methods, we’re adding a CancellationToken parameter. Many users won’t even notice this.

However, we are not breaking the IHandleMessages<T> interface.

Anywhere you’re handling a message ² the existing context object will include a CancellationToken property rather than adding a separate argument that would result in a breaking change.

The built-in CA2016 analyzer provided by the Roslyn team can’t tell you to forward context.CancellationToken to all the other code you call from your message handlers ³, so how do we deal with that?

We’ve had great success bundling our own Roslyn analyzers with NServiceBus, ⁴ and we can use the same strategy in this case.

In NServiceBus version 8, if you call a method that accepts a CancellationToken parameter, you’ll now get a compilation warning which takes the place of CA2016:

Aside from avoiding a breaking change—which is already a big win—we think this has a lot of other advantages.

The default severity for CA2016 is Suggestion. This means that in Visual Studio, you only see three tiny gray dots that are easy to miss, and the message itself is often hidden away in the Messages pane of the Error List window, where it’s often ignored. ⁵

By making our analyzer a warning, we’re elevating the concept of cancellation where it’s more noticeable. We hope that after learning about cancellation, you’ll also use an .editorconfig file to upgrade CA2016 to a warning as well:

[*.cs]dotnet_diagnostic.CA2016.severity = warning

Of course, if your project files use TreatWarningsAsErrors this will prevent code that doesn’t pass the cancellation token from compiling. Or, maybe cancellation isn’t really a concern for your system, and you don’t want to deal with passing cancellation tokens everywhere. That’s fine too! Roslyn analyzers are configurable, so you can also downgrade our analyzer to suggestion or even silent or none if you wish: ⁶

[*.cs]dotnet_diagnostic.NSB0002.severity = suggestion

🔗Summary

NServiceBus version 8 supports cooperative cancellation, which gives you the ability to ensure a message endpoint can shut down quickly and cleanly.

While this feature is critical for many developers, for others, it’s not. That’s why we’ve minimized the breaking changes required to support cancellation, especially on IHandleMessages<T>, our most-used interface.

There is a lot of productivity to be gained by using Roslyn analyzers, and we hope that our new analyzer will help you implement cancellation efficiently and show how analyzers can improve your development workflow.

NServiceBus 8 is available now on NuGet.

But the outbox can only be used inside a message handler. What about web applications and APIs?

With the new NServiceBus.TransactionalSession package, you can use the outbox pattern outside of a message handler too.

🔗The problem

Let’s say you have a web application where you need to create an entity and perform some background processing.

Before the transactional session, the best guidance was to not do any database work inside the ApiController, but to only take the input data and send a message to the back end, responding only with an HTTP 202 Accepted message. Then, a few milliseconds later, a message handler would pick up the message and process it with the complete protection of the outbox feature.

But this isn’t always very realistic. What if the database is in charge of ID generation, and you must return that ID to the client? Or do you need to update the UI to show the request, even if the processing isn’t complete yet?

🔗Ghost protocol

This example code compromises by inserting a single record using Entity Framework and then sends a message to the backend. As a result of the compromise, this code is still vulnerable to ghost messages if the database transaction has to roll back after the message has been sent.

[ApiController]public class SendMessageController : Controller{    readonly MyDataContext dataContext;    readonly IMessageSession messageSession;    public SendMessageController(IMessageSession messageSession, MyDataContext dataContext)    {        this.messageSession = messageSession;        this.dataContext = dataContext;    }    [HttpPost]    public async Task<string> Post(Guid id)    {        await dataContext.MyEntities.AddAsync(new MyEntity { Id = id, Processed = false });        var message = new MyMessage { EntityId = id };        await messageSession.SendLocal(message);        return $"Message with entity ID '{id}' sent to endpoint";    }}

Those familiar with Entity Framework might wonder where the call to SaveChangesAsync is happening. Because Entity Framework already supports the Unit of Work pattern, the calls to SaveChangesAsync can be done using ASP.NET Core middleware, which means the database is only updated when the HTTP request successfully completes:

public class UnitOfWorkMiddleware{    readonly RequestDelegate next;    public UnitOfWorkMiddleware(RequestDelegate next, MyDataContext dataContext)    {        this.next = next;    }    public async Task InvokeAsync(HttpContext httpContext, ITransactionalSession session)    {        await next(httpContext);        await dataContext.SaveChangesAsync();    }}

Regardless of the outcome of the database operations, the moment our controller logic calls SendLocal, the message is handed over to the transport. This makes the message MyMessage almost immediately available to be processed in the background.

Things will often seem fine until a transient exception forces your Entity Framework operations to roll back. The database entity was never committed, but the ghost message starts processing in the background anyway. The message handler tries to load the missing entity from the database…and disaster unfolds.

Imagine looking into this error and trying to figure out why that entity doesn’t exist. Obviously, the message was sent, so the entity should be in the database, but it’s not. Where is it?

Unfortunately, ghost messages don’t know they’re ghosts, ² which makes them hard to diagnose.

🔗Ghostbusters

So how do we banish the ghost message? A proton pack won’t help you, but our new TransactionalSession packages can.

In this case, since we’re using Entity Framework, we’ll use NServiceBus.Persistence.Sql.TransactionalSession, but we’ve got a bunch of different packages available depending on what database you’re using.

In our code, we replace IMessageSession with ITransactionalSession like this:

[ApiController]public class SendMessageController : Controller{    readonly MyDataContext dataContext;-   readonly IMessageSession messageSession;+   readonly ITransactionalSession messageSession;-   public SendMessageController(IMessageSession messageSession, MyDataContext dataContext)+   public SendMessageController(ITransactionalSession messageSession, MyDataContext dataContext)    {        this.messageSession = messageSession;        this.dataContext = dataContext;    }    // Rest omitted. Believe us, the code looks the same ;)}

All the business logic stays the same. With that small change, the newly stored entity and the sends are wrapped in an all-or-nothing transaction, meaning both succeed or fail. The TransactionalSession package works with all the persistence technologies supported by NServiceBus, like Microsoft SQL Server, CosmosDB, and many more.

Check out this video demo of the TransactionalSession in action:

🔗One final detail

Much like the code that ensures Entity Framework operations are executed against the database by calling SaveChangesAsync, there needs to be a tiny bit of middleware that ensures the transactional session is committed when the HTTP pipeline has to be executed.

public class UnitOfWorkMiddleware{    readonly RequestDelegate next;+   readonly ITransactionalSession messageSession;-   public UnitOfWorkMiddleware(RequestDelegate next, MyDataContext dataContext)+   public UnitOfWorkMiddleware(RequestDelegate next, ITransactionalSession messageSession)    {        this.next = next;+       this.messageSession = messageSession;    }    public async Task InvokeAsync(HttpContext httpContext, ITransactionalSession session)    {+       await session.Open(new SqlPersistenceOpenSessionOptions());        await next(httpContext);-       await dataContext.SaveChangesAsync();+       await session.Commit();    }}

Notice how the middleware no longer calls SaveChangesAsync anymore–instead, the middleware commits the transaction on the data context. The transaction created by the transactional session will take care of saving all database changes and triggering the outbox so that everything remains consistent and atomic.

🔗Bulletproof

The algorithm behind the Transactional Session feature was based on the proven NServiceBus Outbox implementation. But we’ve also modeled and verified the algorithm using TLA+, ³ a formal specification language to verify and test programs. Plus, we’ve covered it with a rich set of automated test suites covering every supported database engine, so you know you can trust it.

🔗Summary

With the new TransactionalSession, there are no more compromises. You don’t have to painfully redesign a web application to move all the data transactions to the backend. Instead, you can update the database and send a message, and be confident that the outbox implementation will prevent ghost messages or zombie records.

To get started, check out the TransactionalSession documentation, including a detailed description of how it works. Or, check out our Using TransactionalSession with Entity Framework and ASP.NET Core sample to see how to use it in your own projects.

Previously, we were able to achieve 10X faster pipeline execution and a 94% reduction in Gen 0 garbage creation by building expression trees at startup and then dynamically compiling them. One of the key learnings of those expression tree adventures is that reducing Gen 0 allocation makes a big difference. The less Gen 0 allocation used, the more speed can be squeezed out of the message handling pipeline, which ultimately means more speed for our users.

A common, but overlooked source of allocations is closure allocations. Originally, our message pipeline had, to use the scientific term, a gazillion of them. By getting rid of the major source of closure allocations in the pipeline, we’ve managed to get another five-fold increase in pipeline execution performance, as well as the complete removal of closure-related Gen 0 garbage creation. In this post, we’ll recap what closures look like, explain how they can be avoided, and show the tricks we’ve applied to the NServiceBus pipeline to get rid of them.

🔗Closures lurking beneath the surface

Closure allocations can be hard to spot. Before the introduction of the Heap Allocation Viewer for Jetbrains Rider or Clr Heap Allocation Analyzer for Visual Studio you’d have to either decompile the code or attach a memory profiler and watch out for various *__DisplayClass*, Action* or Func* allocations.

Closures can occur anywhere we have lambdas (i.e. Action or Func delegates) being invoked that access state that exists outside the lambda. Here’s an example:

static void MyFunction(Action action) => action();int myNumber = 42;MyFunction(() => Console.WriteLine(myNumber));

Here is the decompiled code for this snippet:

<>c__DisplayClass0_0 <>c__DisplayClass0_ = new <>c__DisplayClass0_0();<>c__DisplayClass0_.myNumber = 42;<Main>g__MyFunction|0_0(new Action(<>c__DisplayClass0_.<Main>b__1));

In the generated code, we can spot two allocations. The first allocation of the type c__DisplayClass0_0 represents the state class that will “host” the number, and a second allocation of type Action to be able to point to some compiler generated method called <>c__DisplayClass0_.<Main>b__1 that will eventually execute our Console.WriteLine.

All that just to print a number to the console. And keep in mind, closure allocations might not only be in your code; they can also occur when you use .NET classes like the ConcurrentDictionary.GetOrAdd.

🔗Bye bye closures

It’s possible to get rid of the closure and thus remove the extra display class and delegate allocation. Starting with .NET 5, we can mark the lambda as static with C# 9. In this way, the compiler will allow only state that is static or available inside the lambda to be accessed; otherwise, it will display an error.

int myNumber = 42;MyFunction(static () => Console.WriteLine(myNumber));error CS8820: A static anonymous function cannot contain a reference to 'myNumber'.

In the simple example, when the state is just a fixed number, the variable can be marked as a constant to get rid of the CS8820 compiler error.

const int myNumber = 42;MyFunction(static () => Console.WriteLine(myNumber));<Main>g__MyFunction|0_0(<>c.<>9__0_1 ?? (<>c.<>9__0_1 = new Action(<>c.<>9.<Main>b__0_1)));

Yet in reality, code is rarely that simple. What if the number comes from actual user input or is generated with random.Next()? When the input is not static the state must be passed into the lambda somehow. By changing the lambda from Action to Action<object> the lambda will accept a state object that can be passed from MyFunction into the action delegate.

static void MyFunction(Action<object> action, object state) => action(state);int myNumber = 42;MyFunction(static state => Console.WriteLine((int)state), myNumber);<Main>g__MyFunction|0_0(<>c.<>9__0_1 ?? (<>c.<>9__0_1 = new Action<object>(<>c.<>9.<Main>b__0_1)), num);IL_0023: box [System.Runtime]System.Int32

While this gets rid of the display class and the function allocation, unfortunately, the usage of object forces the compiler to emit a box statement. That is, the compiler will box the integer to an object which causes an unnecessary allocation to occur. Whenever possible, the state-based overloads should use generics instead.

static void MyFunction<T>(Action<T> action, T state) => action(state);int myNumber = 42;MyFunction(static number => Console.WriteLine(number), myNumber);<Main>g__MyFunction|0_0(<>c.<>9__0_1 ?? (<>c.<>9__0_1 = new Action<int>(<>c.<>9.<Main>b__0_1)), state);

By ensuring the delegates have all their state available inside the closure and by using C# 9 static lambda support, we no longer allocate unintentionally.

Let’s see how we can apply this to the NServiceBus pipeline.

🔗The state captured in the pipeline

At its core, the NServiceBus pipeline is just a series of delegates of the shape Task Invoke(TInContext context, Func<TOutContext, Task> next) that get chained together to build the pipeline. By design, the state that is required for the execution of the pipeline is passed into the invocation as TInContext and passed out of the invocation into the next part of the pipeline as TOutContext. For example, the incoming context captures information of the received message (such as the headers), as well as the child service provider that resolves dependencies scoped to the execution of the pipeline, and more. Since the context object captures all the important states, there shouldn’t be any closure allocations happening. Let’s verify that by looking at a simple pipeline.

var behavior1 = new MyBehavior1();var behavior2 = new MyBehavior2();var behavior3 = new MyBehavior3();var context = new RootContext();await behavior1.Invoke(context, ctx1 => behavior2.Invoke(ctx1, ctx2 => behavior3.Invoke(ctx2, ctx3 => Task.CompletedTask)));

The context is nicely flowing from one execution to the other. However, the lambda needs to access the behavior instance, so that it can call the Invoke method. If we mark these lambdas as static as shown before, we immediately get the familiar CS8820 compiler error.

await behavior1.Invoke(context, static ctx1 => behavior2.Invoke(ctx1, static ctx2 => behavior3.Invoke(ctx2, static ctx3 => Task.CompletedTask)));error CS8820: A static anonymous function cannot contain a reference to 'behavior2'.error CS8820: A static anonymous function cannot contain a reference to 'behavior3'.

The compiler error shows us that the NServiceBus pipeline captures the behaviors that are taking part in the pipeline execution as state inside the lambda, which causes closure allocations. To solve this, we need a way to bring the behaviors for each part of the pipeline into the pipeline itself.

🔗Make the behaviors part of the pipeline state

The idea is simple but powerful. Besides being a series of delegates, the pipeline is also essentially a collection of behaviors. So if we can somehow store that collection of behaviors in the context of the pipeline, we’re all set.

The good thing is that once the pipeline execution plan is built, the order of the behaviors is well-known and never changes. So the collection representing all the behaviors including their order can be created once and reused over and over again. Luckily, all behaviors inside the pipeline implement a non-generic marker interface: IBehavior. With that in mind, we can store all behaviors of the pipeline in an array of IBehavior objects in the context.

public class BehaviorContext : IBehaviorContext {  internal IBehavior[] Behaviors { get; set; }}// done once at initialization/startup timevar behavior1 = new MyBehavior1();var behavior2 = new MyBehavior2();var behavior3 = new MyBehavior3();var cachedPipeline = new IBehavior[3] { behavior1, behavior2, behavior3 };var context = new RootContext();// assigned every time a new context is builtcontext.Behaviors = cachedPipeline;

Now that all the behaviors can be accessed from the context, the pipeline execution plan builder can build the pipeline invocation plan that will access the behaviors array at a specific index, cast the returned behavior to the right behavior type, and then call Invoke:

await behavior1.Invoke(context, static ctx1 => ((MyBehavior2)ctx1.Behaviors[1]).Invoke(ctx1, static ctx2 => ((MyBehavior3)ctx2.Behaviors[2]).Invoke(ctx2, static ctx3 => Task.CompletedTask)));

That’s it! No more closure allocations. With this simple change, we’ve seen 5 times more throughput in the raw pipeline execution benchmarks and all allocations that previously occurred due to closures are gone.

🔗Summary

Closure allocations can occur in various places whenever there are delegates involved that are accessing state outside the closure (or the curly braces) of the lambda. By removing closure allocations on code that is executed at scale thousands of times per second, it is possible to achieve significant gains in terms of throughput. Since the NServiceBus pipeline is such a core part of our framework, it has to be solid and fast. So we continue to make small but impactful performance improvements to NServiceBus. Hopefully this deeper understand of closure allocations can improve your own code as well.

But removing closure allocations to achieve high-performing and low-allocating code is just one trick in your tool belt. In my recent webinar “Performance tricks I learned from contributing to open source .NET packages” I summarized not only this tip, but others that I’ve learned from contributing over fifty pull requests to the Azure .NET SDK.

If you are interested in hearing more technical details about other improvements, leave a comment here or reach out to me and I’ll tell you how we managed to get another 11-23% throughput improvement in the NServiceBus pipeline execution, or how we got rid of all allocations from the saga persistence deterministic ID creation. Until then, stay safe and allocation free.

But to fully answer these questions, it’s essential to understand what Kafka is, and more importantly what it isn’t, and then think about the kinds of problems that Kafka solves. So, let’s dive into the (heavily footnoted) details…

🔗What is Kafka?

Apache Kafka is a partitioned log and is a similar product in architecture to Azure Event Hubs or Amazon Kinesis. A partitioned log works precisely as its name suggests: it’s like a bunch of separate log files (partitions) constantly writing whatever data you throw at them. Those bits of data are usually called events, ¹ and one of these append-only log files is called an event stream.

Events are written to an event stream in the order in which they are received, and read in the same order as well. ^{2 3} The events will then stay in that log until a defined retention period is reached—nothing else other than the passage of time can remove an event from the log once it is written.

When you want to receive an event from Kafka, you need to decide how you will read from the event stream. You are responsible (or, more accurately, the client library you are using is responsible) for determining what partition you are reading from and keeping track of your position within that partition. So you don’t just get the next message waiting to be processed. Instead, you have to decide where you want to start reading from (called a “cursor”) and then keep that cursor up-to-date yourself.

If another reader wants to receive an event, they can’t just pick up where you left off. Instead, they have to manage their own cursor because Kafka doesn’t manage cursors for you. It just provides you with repeatable access to the stream of events.

While Kafka can deal with high traffic volumes, the right partition strategy is a key factor and will have a major impact on the scalability and elasticity of the whole system. Choosing the right number of partitions for event streams is a bit of an art and can be difficult to change, especially on the fly.

But what happens if something fails while processing an event in Kafka? Since the log stream is still all there, ⁴ you can just re-read the log stream from a specific offset or decide to skip over the offset of the troublesome event altogether.

So now we know more about Kafka, but the big question remains, is it a message queue?

🔗What are message queues?

A message queue or message broker is a highly-optimized database for processing messages.

Like Kafka, events (but here, let’s call them messages) can be stored in the same order in which they are received. But that’s where the similarities end.

The goal of a message queue is to be empty. Therefore, it does not have a retention period; it only hangs on to a message until a client confirms it has been successfully processed, and then that message is deleted from the broker.

Each message queue client is not responsible for managing a cursor, as that is managed centrally by the broker. Therefore, the only message you can receive is the one that is next on the queue. If you attempt to process that message, the broker will prevent any other consumer from getting access to the same message unless you report you were unable to successfully process it ⁵ or a timeout expires. When this happens, the broker is forced to assign the message to someone else.

This behavior leads to the competing consumers pattern, where multiple clients can cooperate to process messages more quickly, something that Kafka and other event streams are not designed to do. Unlike Kafka, consumers can be added and removed at any point in time without any impact on the topology of the queuing system.

The error handling pattern on message queues is also different. With a queue, you only get access to the next message in the queue, but sometimes, that message can’t be successfully processed. For example, it may not be possible to deserialize its contents. When that happens, that message effectively blocks the processing of subsequent messages. Messages which cannot be successfully processed are called poison messages. Systems that rely on a message queue have a process for retrying messages and eventually will forward poison messages to an error queue or dead-letter queue for investigation. ⁶ Kafka can be made to emulate some of these error-handling patterns, but this will require more work than message queues, where it’s the default behavior. ⁷

For an even deeper dive into message queues, including how you’re using multiple queues right now without even realizing it, check out this video by Clemens Vasters, Principal Architect for the Microsoft Azure Messaging platform:

So as we can see, the only thing that partitioned logs (like Kafka) and message queues (like RabbitMQ, Azure Service Bus, and Amazon SQS) have in common is that they help you process messages. But, after that, the differences couldn’t be more stark.

🔗So which is better?

Now that we’ve established that Kafka is not a message queue, ⁸ the main question that remains is which is better?

The answer, of course, is neither. There are good reasons to use both, even at the same time.

🔗When to use Kafka

Kafka is great for situations where you need to ingest or process large amounts of data, and you want to be able to read that data repeatedly, typically from different logical consumers. This is very common in systems for telemetry and data distribution.

Consider using Kafka when “events” represent the state of something at a specific time, but any individual event doesn’t have much business meaning by itself. In these cases, you need a stream of these events to analyze changes and transitions in state to derive any business meaning.

🔗When to use a message queue

Message queues excel when messages must be successfully processed once and only once, and losing data is not an option. These systems depend on reliable state transition management for business processes.

When using a message queue, messages represent information about a specific business process step. Therefore, every message in a queue has business value by itself; it doesn’t necessarily need to be analyzed in relation to other messages.

Because every single message has significance, messages have to be processed independently and can be scaled using the competing consumers pattern. After a given message has been successfully processed, it’s no longer available to any consumers.

🔗When to use Kafka AND message queues

Imagine a system that monitors changes to stock prices and alerts users when specific changes occur.

Consuming the firehose of real-time stock price data is a perfect job for Kafka, which excels at handling that amount of data and storing it for further processing. Each point-in-time stock price value isn’t that useful to us; it’s just the changes we’re interested in.

There may be many different consumers of this data. Of course, there may be consumers that are interested in different stocks. Depending on the system requirements, the streams may be set up so that each stock’s data lives on separate partitions, and any reader is only looking at the values for one stock symbol at a time. Or, stocks may be organized on the same partition, and readers simply ignore data they don’t care about.

However, even for one stock symbol, we may have many readers looking at the same event stream but interested in different things. For example, one reader may be interested only in sudden spikes or drops in a stock price, while another may be monitoring for trends over an extended period.

It ultimately doesn’t matter because each reader of a partitioned log maintains its own cursor and can read the same stream as often as it wants.

Once one of these Kafka stream readers detects an important business event, such as a sudden increase in stock price, that’s the time to use a message queue. Now we publish a StockPriceSpikeDetected event using a message queue, so that message handlers can execute business processes for that event. These message handlers might be making stock trades, updating data in a database, emailing fund managers, sending push notifications to mobile applications, or whatever else needs to be done.

It can be helpful to think of Kafka events more as raw data. Only when trends in the data become relevant to the business does something become a business event, and that’s the point at which you should be using a message queue. In fact, we’ve seen customers use Kafka in this way. They host code in Azure Functions using a KafkaTrigger to monitor event streams, then raise business events in Azure Service Bus (using either a send-only NServiceBus endpoint or native sends) which are processed by NServiceBus endpoints. That’s a more appropriate use case for Kafka, and it works great.

🔗And what about NServiceBus?

Here at Particular Software, we believe in using the right tool for the job. So while it is possible to employ complex workarounds to make Kafka kinda-sorta behave somewhat like a queue, we think that if you have a situation where you should use a message queue, you should use a message queue.

Since NServiceBus is a communication framework over message queues, we don’t currently have any plans for a Kafka-based message transport that would be similar to our message transports for RabbitMQ, Azure Service Bus, and Amazon SQS. Unfortunately, when implementing a message transport, the devil is in the details—many of the features our customers love most about NServiceBus are much harder to achieve with Kafka due to seemingly inconspicuous differences between message queues and event stream processing infrastructure.

However, we are investigating other communication abstractions that might suit Kafka (and other partitioned logs) better. And that’s where you come in.

If you’re interested in using Kafka (or another partitioned log) we want to hear from you. Drop us a line and tell us what you want to do. We’re far more interested in building the right thing than getting something out there to check a Kafka box for the folks in the marketing department.

So let us know what you’re up to—we’d love to chat.

• manage messages that require a manual retry
• see when endpoints go offline and back online
• detect connectivity problems with databases, brokers, and other systems using custom checks
• troubleshoot message processing performance, both per-message and across an entire flow, using ServiceInsight

Many of these capabilities have been developed over time as separate plugin packages. Each one has brought its own code-first configuration API. This approach has led to a situation where getting the most from the platform means installing six different NuGet packages and calling seven different configuration APIs.

We knew we could make it easier. What if it was just one package, and one configuration API?

🔗One package, one API

The new NServiceBus.ServicePlatform.Connector package simplifies connecting an NServiceBus endpoint to the Particular Service Platform by putting all of the configuration details into an easily serialized set of classes. It also includes a simple API to apply those configuration details to an endpoint.

var json = File.ReadAllText(pathToConfiguration);var platformConnection = ServicePlatformConnectionDetails.Parse(json);endpointConfiguration.ConnectToServicePlatform(platformConnection);

The configuration file is in JSON format:

{  "ErrorQueue": "error",  "MessageAudit": {    "Enabled": true,    "AuditQueue": "audit"  },  "SagaAudit": {    "Enabled": true,    "SagaAuditQueue": "audit"  },  "Heartbeats": {    "Enabled": true,    "HeartbeatsQueue": "Particular.ServiceControl",    "Frequency": "00:00:30"  },  "CustomChecks": {    "Enabled": true,    "CustomChecksQueue": "Particular.ServiceControl"  },  "Metrics": {    "Enabled": true,    "MetricsQueue": "Particular.Monitoring",    "Interval": "00:00:05",    "InstanceId": "MyEndpointUniqueInstanceId"  }}

With this file in place, you can enable and disable features and adjust configuration at deployment time in each environment.

“But that’s just moving the complexity to a file,” you astutely point out. If you have to hand-craft the file then we haven’t really simplified anything. We thought of that too.

With the latest versions of ServiceControl and ServicePulse, you can view the JSON configuration directly in ServicePulse.

You don’t have to store the configuration in a file, or even as JSON. You can integrate with anything that can handle strongly-typed configuration. Here’s a sample showing how to retrieve ServicePlatform Configuration from Microsoft.Extensions.Configuration.

🔗Summary

Centralized configuration makes things much easier for anyone who is managing a distributed system. With the NServiceBus.ServicePlatform.Connector we’ve greatly simplified this process for ServiceControl and we think you’ll agree it’s a better experience.

Get the latest versions of ServiceControl and ServicePulse here:

• ServiceControl releases
• ServicePulse releases

In version 1.1 of our CosmosDB persistence package, we’ve made defining the partition key for each message processed by NServiceBus much more straightforward, without needing a custom pipeline behavior. We’ve also added pessimistic concurrency support for more reliable processing of sagas with high contention patterns.

Let’s take a closer look at these two new features.

🔗Partition configuration API

We made it a lot easier to specify the container partition to use for each message, which is essential to make Cosmos DB transactions work.

NServiceBus uses Cosmos DB transactions to keep NServiceBus outbox and saga data consistent with whatever business data you modify in your message handlers. Cosmos DB supports transactions through its TransactionalBatch API in the .NET SDK, and NServiceBus gives you access to the TransactionalBatch so that you can use it for your business data.

There’s just one catch: all the operations in the transaction must take place in the same partition within a container. So, for each incoming message, you must tell NServiceBus which container partition to use so that the NServiceBus data and your business data can be stored together.

Previously, specifying the partition key required implementing a custom pipeline behavior to provide the information needed for the transaction. A pipeline behavior is an advanced NServiceBus API, which is very powerful, ¹ and there are a lot of good reasons to use one, but you shouldn’t have to create one just to use Cosmos DB.

We made this process more straightforward with a new transaction information API that allows you to provide NServiceBus with the necessary information without poking under the hood.

Here are a few examples of how to use the new API:

// Get the configuration objects we needvar persistence = endpointConfiguration.UsePersistence<CosmosDBPersistence>();var transactionsInfo = persistence.TransactionInformation();// The partition to use is always located in a message headertransactionsInfo.ExtractPartitionKeyFromHeader("PartitionKeyHeader");// OR you can use multiple headerstransactionsInfo.ExtractPartitionKeyFromHeaders(headers => new PartitionKey(…));// OR get the partition key from the messagetransactionsInfo.ExtractPartitionKeyFromMessage<MyMessage>(message => new PartitionKey(message.PartitionKey));// OR use a custom class that implements IPartitionKeyFromHeadersExtractortransactionsInfo.ExtractPartitionKeyFromHeaders(new CustomPartitionKeyFromHeadersExtractor());

There are a lot of options to cover a variety of use cases, all of which are much easier than defining your own NServiceBus pipeline behavior. Check out the documentation for more API options for advanced scenarios.

This is a much easier way to configure NServiceBus to use your tenant-per-container or tenant-per-partition scheme. Even if you aren’t building multi-tenant systems, the new configuration API makes it easier to align your NServiceBus processing with your chosen partitioning scheme. No more tinkering with the internals of NServiceBus. ²

To learn more about building multi-tenant systems with NServiceBus and Cosmos DB and how to design your data partitioning strategy to fit your requirements, check out our recent webinar:

🔗Pessimistic concurrency support

One of the most powerful features of an NServiceBus saga is how it handles multiple messages trying to modify the same data simultaneously. No matter what, the saga will ensure that two concurrent messages can’t make conflicting changes to the stored saga data that would result in a corrupted state.

However, how the saga controls access impacts the system performance and cost to run the system under certain conditions.

The original version of Cosmos DB persistence supported only optimistic concurrency. In this strategy, message handlers for multiple messages can start processing concurrently, but the first one to commit their changes wins. When other message handlers try to commit, they get a concurrency exception (because the underlying data has changed) and are forced to retry.

This works well for sagas with little or no contention, and the performance is good. From the Cosmos DB perspective, this is also the cheapest option because you don’t have to perform any database operations (which cost money) to determine if it’s safe to proceed.

However, some sagas, such as those that implement the scatter-gather pattern, have much higher contention, and that’s when optimistic concurrency starts to break down. Many competing messages cause many concurrency exceptions to be thrown when the first message commits, resulting in floods of retries that increase the overall load, decrease message throughput, and may result in many failed messages in the error queue. ³

For sagas with high contention, pessimistic concurrency is a better approach. In this mode, we don’t try to process the message until a lock has been acquired so that we’re sure when starting the message handler that we’ll be able to commit the changes later. Every other message that needs access to the same saga data must wait until the lock is released. Then, it can obtain a new lock and proceed with processing.

This method results in fewer failures and eases contention, especially in scatter-gather scenarios, but comes at a cost. Because Cosmos DB charges for each storage operation, there is increased cost associated with checking for and obtaining the lock before a message is processed. Additionally, sagas normally unaffected by contention issues will now process more slowly due to the extra locking behavior.

Because of the extra cost associated with pessimistic concurrency, it’s not enabled by default. To enable it:

var persistence = endpointConfiguration.UsePersistence<CosmosDBPersistence>();persistence.Sagas().UsePessimisticLocking();

We recommend only enabling pessimistic locking in endpoints that contain sagas prone to contention issues. All other endpoints can use the default optimistic locking strategy.

Check out the Cosmos DB persistence documentation page for saga concurrency for more details on how to use and tune pessimistic locking to get the best out of your endpoints with high-contention sagas.

🔗Summary

With Cosmos DB persistence version 1.1, it’s even easier to create a Cosmos DB system, align it to your partitioning scheme, and then manage its performance.

To learn more about Cosmos DB and NServiceBus, check out our Cosmos DB persistence documentation. If you’re currently using Azure Table Storage in your system, check out how to migrate from Azure Table storage to Cosmos DB. We’ve also got several code samples showing how to use Cosmos DB with NServiceBus.

We’ve focused on supercharging your ability to develop NServiceBus sagas in our latest round of releases. As a result, you’re going to feel like you’ve got your own “heads-up display” when developing sagas. We’ll give you suggestions and point out problems before you even hit compile. You focus on your business logic.

Let’s take a look at the new features.

🔗Saga analyzers

Sagas have a powerful API, but it’s limited by the confines of C#. You can create a class that is a lousy saga but is still perfectly valid C# code. With Roslyn analyzers and code fixes, we can now do a lot better and help guide you toward the pit of success.

In NServiceBus version 7.7, we’ve added a variety of Roslyn analyzers that distill many of our saga development best practices and make them available as hints in your Error List window and as red squiggles ² directly in your code.

Some of the diagnostics created by the new analyzers simply elevate a runtime error to compile time. This will save you time, as you won’t have to run your code to find out that you can’t use DateTime as a correlation property type.

Others will save you time by doing some of the coding for you. Check out how you can now generate the code for the ConfigureHowToFindSaga method when adding an IAmStartedByMessages<T> to your saga:

That’s for a new saga, where the correlation id OrderId hasn’t been defined yet. Watch what happens when we add another message OrderBilled that already has a matching OrderId property:

Since the analyzer already knows that the saga data’s OrderId property is the correlation id, and the OrderBilled message also contains an OrderId property, the generated code already includes the correct mapping, and there’s nothing more to do.

We created a bunch of analyzer diagnostics—15 in total—to supercharge your saga development. Check out Roslyn analyzers for sagas in our documentation for all the details.

🔗Saga scenario testing

While we were making it easier to write sagas, we thought it was also essential to make it easier to test them. So the newest version of our testing framework now includes tools to perform saga scenario testing, which are more expressive than testing sagas with standard unit tests.

For a long time, we’ve had the NServiceBus.Testing package which provided a means to write unit tests on message handlers, both regular handlers and those found inside sagas.

This has always worked pretty well for regular message handlers, but it has always seemed a bit clunky for testing sagas. You have to understand too much about how sagas work internally to write effective and accurate unit tests, or you can miss important details. For example, did you know that NServiceBus will automatically assign the saga data’s correlation property value with the value from the first incoming message? How about that an external message handler replying to a saga will include the SagaId in a message header, which means a mapping in the ConfigureHowToFindSaga method isn’t required? These details can really trip you up, leading to tests that don’t test what you think they are.

There was also no way to test the ConfigureHowToFindSaga method. At all. Unit tests have to call the saga’s Handle methods directly, without exercising any mapping expressions inside the ConfigureHowToFindSaga method, so you just had to hope really hard that your mappings were all correct.

With our new saga scenario testing framework, you can now do more than make assertions on the result of one message handler. Instead, the framework enables testing the result of a whole series of messages (a scenario) at a time, where handling each message exercises the ConfigureHowToFindSaga mappings to load the saga data from a virtual data store managed by the test.

This enables tests like “Is the OrderShipped event published after the OrderPlaced and OrderBilled are received, and the buyer’s remorse period has ended?” or “If the messages arrive in a different order than I expect, will the order still be shipped?”

The testable saga also keeps track of a virtual CurrentTime, stores saga timeouts internally, and plays them only when you call the AdvanceTime method. This gives you complete control over the scenario and enables testing race conditions—what happens if a specific message arrives before a timeout fires, or the other way around?

We know this new testing framework will make your sagas easier to test. Additionally, we hope that the tests you write with it will be more expressive. With scenario testing, each test can tell a complete story that documents the behavior you expect the saga to exhibit.

🔗Critical time metric

In NServiceBus version 7.7, we’re adding information to outgoing messages to more accurately calculate the critical time metric, which tells you how long after a message is sent for it to be fully processed. This metric is essential to ensure that you meet your message processing SLAs and can also be used as a trigger to scale out infrastructure when it becomes apparent that the SLA will be breached.

We realized that critical time had a serious flaw when it came to sagas: delayed messages caused by saga timeouts include the delay in addition to the delivery and processing time. So if you had saga timeouts from a year ago (which is a normal part of some business processes!), it would look like your critical time was one year, when there’s actually nothing wrong with your system performance.

To fix this problem, a new DeliverAt message header has been added to outgoing messages that include a delay to calculate the critical time attribute more accurately.

We also updated the critical time calculation in NServiceBus.Metrics version 3.1 to use this new information. If you update your system with NServiceBus 7.7 and NServiceBus.Metrics 3.1, the performance metrics in ServicePulse will reflect the new method of calculating critical time information.

A future release of ServiceControl will update how critical time is displayed for audit messages in ServiceInsight.

🔗Saga not found logging

One small (but important!) way we’re always trying to improve NServiceBus is through our logging and error messages. We want to make sure the exception and log messages we put in front of you are clear and let you know precisely what you need to do. So, as part of NServiceBus 7.7, we adjusted the logging that occurs when saga data is not found.

If a message is handled by a saga but does not start the saga, NServiceBus will log a message. Of course, it’s possible the saga has done its job and doesn’t care about that type of message anymore, so it’s not necessarily a problem—but it could be.

Previously the log message indicated that a saga was not found for a specific message type. However, the log would not include the type of saga that was not found. In cases where a message was handled by multiple sagas, the message would be shown only if saga data couldn’t be found for any of them.

In NServiceBus 7.7, we’ve made this a lot clearer. For each saga type where the data could not be found, we now log:

Could not find a started saga of ShippingPolicy for message type OrderPlaced.

And if all sagas that handle a message are not found, we log:

Could not find any started sagas for message type OrderPlaced. Going to invoke SagaNotFoundHandlers.

This additional saga type information in the log should provide more clarity when investigating these types of situations.

🔗Summary

We’re always trying to make development with NServiceBus simpler, easier, and more powerful. Whether it’s our new saga analyzers, our new scenario testing framework, or our fixes for critical time and saga not found logging, we hope you’ll find something that will supercharge your own saga development.

It may not be an actual video game heads-up display, but we do what we can.

To get all these updates, you’ll want to update to NServiceBus 7.7.0, NServiceBus.Testing 7.4.0, and NServiceBus.Metrics 3.1.0. Keep in mind that you might be using NServiceBus.Metrics as a transitive dependency of NServiceBus.Metrics.ServiceControl or NServiceBus.ServicePlatform.Connector, and because NuGet will only load the lowest matching version of a transitive dependency, you will need to add an explicit package reference to NServiceBus.Metrics 3.1.0 to get the critical time update.

Distributed systems can “get frustrated” too. One tiny thing goes wrong, and suddenly every message starts to fail. If you’re lucky, it just throws a bunch of errors. If you’re unlucky, it goes into a tight loop of failure that results in a hefty bill from your cloud provider.

🔗Failure on repeat

Distributed software systems built with NServiceBus are pretty great about dealing with all kinds of failure. From transient to systemic exceptions and everything in-between, systems built on reliably exchanging messages are resistant to failure, but that doesn’t prevent issues from occurring.

If your message handler relies on a 3rd-party service, and that service is not available, NServiceBus will keep retrying that message and won’t lose the data contained in the message. Eventually, that message gets sent to an error queue. And if you have high message throughput, you might have a lot of messages headed for an error queue, all of which are going to need to be replayed later.

In cloud environments, this can be especially problematic. (Read: expensive!) Every single attempt to process a message costs money, which is silly when we already know trying to process messages is going to be futile until the 3rd-party service is available again. You literally have to pay money to accomplish nothing.

🔗What to do with consecutive failures

NServiceBus 7.6 now tracks the number of consecutive failures and lets you take action to minimize its effect on the system.

For example, if your message handlers are all calling an unavailable 3rd-party service, all messages will likely fail. After, say, 10 consecutive failures, it should be clear that something more serious is going on. ¹

Now you can change how messages are processed to prevent flooding the error queue. After enough consecutive failures, you can now enter a throttled mode where NServiceBus will only attempt to process one message at a time at a rate you specify.

It’s just like pushing pause in a video game and walking away. Instead of trying over and over as fast as possible, the endpoint walks away for a while. Instead of sending every message to the error queue, the system attempts one message per second to see if the situation has improved.

It doesn’t have to be a long wait—trying one message every few seconds usually works pretty well. The critical point is that when the system becomes fully operational again, there are only a handful of messages in the error queue instead of hundreds or thousands.

And if you’re in the cloud, the system didn’t just spend a small fortune chasing its tail.

🔗Rate limiting on consecutive failures

To enable the throttled one-message-at-a-time processing mode, we have introduced an API on the RecoverabilitySettings class:

var recoverability = endpointConfiguration.Recoverability();recoverability.OnConsecutiveFailures(10,  new RateLimitSettings(    timeToWaitBetweenThrottledAttempts: TimeSpan.FromSeconds(1),    onRateLimitStarted: () => Console.Out.WriteLineAsync("Rate limiting started"),    onRateLimitEnded: () => Console.Out.WriteLineAsync("Rate limiting stopped")));

With this setting, the endpoint will switch to a rate-limited mode after it experiences 10 consecutive failures. By default, this mode will change the endpoint concurrency to 1 and wait 1 second after each attempt. However, as soon as a single message is processed successfully, the endpoint will revert to the regular processing mode with the previous concurrency setting and no delay after attempts.

The RateLimitSettings class allows you to configure the delay between processing attempts and take action when rate-limiting starts and stops.

The exact settings you use depend on the circumstances for each endpoint. How many consecutive failures should determine a persistent failure state? And how often do you want to check to see if things have improved? That’s up to you.

🔗Summary

When you get frustrated in a video game, sometimes the best thing is to pause the game and walk away. Then, a bit later, you come back more relaxed, pick up the controller, and nail it on the first try.

NServiceBus 7.6 lets you do the same thing with your distributed system. Instead of going into “failure on repeat” mode and generating a big cloud resource bill, NServiceBus can now notice the repeated failures and push pause, patiently waiting until conditions improve, and then it’s back to normal.

NServiceBus 7.6 is available now. You can download NServiceBus 7.6 from NuGet, read the release notes, or check out the automatic rate-limiting documentation.

Just remember it’s healthy to step away once in a while.

“But isn’t RPC faster than messaging?”

In place of RPC, ¹ they may substitute a different term or technology like REST, microservices, gRPC, WCF, Java RMI, etc. However, no matter the specific word used, the meaning is the same: remote method calls over HTTP. So we’ll just use “RPC” for short.

Some will claim that any type of RPC communication ends up being faster (meaning it has lower latency) than any equivalent invocation using asynchronous messaging. But the answer isn’t that simple. It’s less of an apples-to-oranges comparison and more like apples-to-orange-sherbet.

Let’s take a look at the bigger picture.

🔗Why RPC is “faster”

It’s tempting to simply write a micro-benchmark test where we issue 1000 requests to a server over HTTP and then repeat the same test with asynchronous messages. But to be fair, we also have to process the messages on the server before we can consider the messaging case to be complete.

If you did such a benchmark, here’s an incomplete picture you might end up with:

Initially, the messaging solution takes longer to complete than RPC. And this makes sense! After all, in the HTTP case, we open a direct socket connection from the client to the server, the server executes its code, and then returns a response on the already-open socket. In the messaging case, we need to send a message, write that message to disk, and then another process needs to pick it up and process it. There are more steps, so the increased latency is easily explained.

But that’s just a micro-benchmark and doesn’t tell you the whole story. Anyone who shows you a graph like that and says “RPC is faster” is either lying or selling you something. ²

🔗Threads and memory

Unfortunately, the web servers serving your RPC request won’t scale linearly forever, which becomes a big problem.

What you typically see nowadays in a “microservices architecture” using RPC ³ is not a single RPC call, but instead, one service calling another service, which calls another service, and so on. Even a service that doesn’t turn around and call another service usually has to do something like talk to a database, which is another form of RPC.

What happens with threads and memory when you’re doing these remote calls?

When you begin a remote call, any memory you had allocated needs to be preserved until you get a response back. You may not even be thinking about this as you’re coding, but whatever variables you’ve declared before the RPC call must retain their values. Otherwise, you won’t be able to use them once you have your response.

Meanwhile, the garbage collector (or whatever manages memory in your runtime environment) is trying to make things efficient by cleaning up memory that’s not used anymore.

Garbage collectors are designed under the assumption that memory should be cleaned up reasonably quickly. So in relatively short order, the garbage collector will perform Generation 0 (Gen0) collection, in which it will ask your thread, “Are you done with that memory yet?” Nope, as it turns out, you’re still waiting for a response from the RPC call. “No problem,” the garbage collector will say, “I’ll come back and check with you later.” And it marks that memory as Generation 1 (Gen1), so it knows not to bother your thread again too soon.

Around ~50,000 CPU operations later, the garbage collector will come around for a Gen1 memory collection. This is a long time in terms of CPU cycles, but it’s maybe about 50 microseconds for us humans, which isn’t much at all. It’s also not a long time in terms of a remote call, which is way slower than a local function execution. “Are you done with that memory now?” Your thread is shocked—doesn’t the garbage collector understand how long remote calls take? “No problem,” the collector says, “I’ll come back later.” And it marks your memory as Gen2.

The actual timings of the garbage collector’s activity will vary on a lot of things, but the point is that your memory can be put into Gen2 before your RPC call even completes. This is important because the garbage collector doesn’t actively clean up Gen2 memory. So even if you get a response back from the server and your method completes, your Gen2 memory may not be cleaned up except for IDisposable objects.

Regular memory just sits there in Gen2. You essentially have a minor memory leak when you’re invoking remote calls if those calls take enough time to come back. This memory accrues until the system is under enough load that it can’t allocate additional memory anymore.

Then the garbage collector says, “Uh-oh, I guess I better do something about this Gen2 memory.”

🔗Stop the world

This is where the throughput of an RPC system starts to go off the rails.

The garbage collector has already tried to clean up Gen2 memory twice, and it’s obviously being actively used by the thread. So the garbage collector’s only choice is to suspend all the currently-executing threads in the process to clean up the Gen2 memory.

That’s when the throughput of your RPC system starts to look like this:

The scale on the Load axis is expanded beyond a micro-benchmark now. As the garbage collector starts suspending the threads of your process to clean up memory, all the clients waiting for a response from you now have to wait longer.

This creates a dangerous domino effect. As one part of your system gets slower, it responds more slowly to its clients, which means their memory goes into Gen2 faster, which means their garbage collector will suspend their threads more frequently, which means their clients must wait longer…you can see where this is going. The deeper your RPC call stacks are from one microservice to the next, the more accumulated memory pressure you have. And the more memory pressure you have, the more likely it is that you will find yourself in this sort of situation:

On the right side of the graph, the process can’t spin up more threads to handle additional incoming requests because it ran out of memory. Meanwhile, you, the client, are receiving the exception “Connection refused by remote host.” You’re getting a response, but the server is saying, “Look, I’m too busy. You’re going to have to come back later.” It can’t afford to spin up more threads and is load shedding, which is the only mechanism an RPC system has to handle the excess load.

If all you have is a single client and a single server, this usually isn’t going to be a big deal. But the more small moving parts that you have, the more fragile the system will be.

🔗Load in messaging systems

Systems built on messaging, under load, will usually exceed the throughput of an RPC-based system.

Systems built on message queues don’t do load shedding like RPC systems because they have storage on disk to store incoming requests as they come in. This makes a queue-based system more resilient under a higher load than an RPC system. Instead of using threads and memory to hold onto requests, it uses durable disks. As a result, many more messages can be sitting in queues even while the system is processing at peak capacity.

This is why it’s like apples and orange sherbet to compare RPC and messaging using a micro-benchmark like at the beginning of this article: If you’re allowed to throw away whatever requests you feel like, it’s not a fair comparison. Messaging doesn’t do that.

In a messaging-based system, there’s usually no waiting around for responses from other microservices. I receive a message, I write something to my database, and maybe I send out additional messages, and then on to the next. Message-based microservices are running very much in parallel with each other. With no waiting, message processing under load will scale up to a certain point, based on a number you configure: the maximum number of concurrently processing messages. ⁴

With all the parallel processing and no waiting, the messaging architecture generally overtakes the RPC under load, resulting in a higher (and more importantly, stable) overall throughput.

How much higher throughput? It depends very much on the system, how you designed it, whether the database is the bottleneck, and a million other factors. But usually, the async processing model results in a more parallel-processing result, resulting in higher throughput for your system than the synchronous blocking RPC model.

🔗Summary

Anytime we use a synchronous RPC model, there’s always a risk of an “epic fail” scenario. There’s always a risk that an RPC system will start running out of threads and memory, the garbage collector will start suspending threads more frequently, the system will do more housekeeping than business work, and soon after, it will fail.

Systems built on asynchronous messaging won’t fail like that. Even if the RPC system doesn’t fail, the messaging sytem will usually exceed the throughput of an RPC system.

If you’d like to learn how to build messaging systems this way, join me for a live webinar Live coding your first NServiceBus system where I’ll show you all the fundamental messaging concepts you need to understand to build an effective distributed system using messaging instead of relying on RPC communication.

Have you heard of C# source generators ¹ and thought they sounded pretty cool but didn’t really know how they could be useful?

In the newest version of our Azure Functions integration, we’ve used source generators to reduce the boilerplate needed to set up an NServiceBus endpoint on Azure Service Bus down to just a few lines of code.

Now, this code is all it takes to write transactionally consistent NServiceBus handlers inside of your Azure Function project.

[assembly: FunctionsStartup(typeof(Startup))][assembly: NServiceBusTriggerFunction("MyEndpoint", SendsAtomicWithReceive = true)]class Startup : FunctionsStartup{    public override void Configure(IFunctionsHostBuilder builder) =>        builder.UseNServiceBus();}

From those attributes, we’ll generate an Azure Function trigger, wire it up to an Azure Service Bus queue, and manage flowing transactions around for you. All you have to do is supply the business logic. Intrigued? Let’s dive in to see what’s new with NServiceBus and Azure Functions.

🔗Automatic trigger function generation

Both NServiceBus and Azure Functions provide abstractions over receiving and handling messages from an Azure Service Bus queue. To get them working together, we need a bit of boilerplate code to create a functions trigger that passes everything needed to NServiceBus. In the 1.0 release, that looked something like this:

using System.Threading.Tasks;using Microsoft.Azure.ServiceBus;using Microsoft.Azure.WebJobs;using Microsoft.Extensions.Logging;using NServiceBus;class FunctionEndpointTrigger{    readonly IFunctionEndpoint endpoint;    public FunctionEndpointTrigger(IFunctionEndpoint endpoint)    {        this.endpoint = endpoint;    }    [FunctionName("NServiceBusFunctionEndpointTrigger-ASBTriggerQueue")]    public async Task Run(        [ServiceBusTrigger(queueName: "ASBTriggerQueue")]        Message message,        ILogger logger,        ExecutionContext executionContext)    {        await endpoint.Process(message, executionContext, logger);    }}

We didn’t like all that boilerplate, so we picked out the important details that need to be configured, and we used source generators to allow you to create an Azure Function trigger that maps to an NServiceBus endpoint with a simple attribute.

[assembly: NServiceBusTriggerFunction("ASBTriggerQueue")]

Dropping this attribute into your Azure Functions project will automatically generate the same common code shown above at compile time. Then, you can delete your custom trigger altogether. We believe that most of the code in your project should be business logic for handling messages.

You can read more about this feature in our documentation.

🔗Consistent messaging

We also used source generators to make it really easy to use the transactional processing in Azure Service Bus that gives you consistency between your incoming and outgoing messages.

Imagine a message handler that looks like this:

class ProcessOrderMessageHandler : IHandleMessages<ProcessOrder>{  public async Task Handle(ProcessOrder message, MessageHandlerContext context)  {    await context.Send(new BillOrder { OrderId = message.OrderId });    await context.Send(new CreateShippingLabel { OrderId = message.OrderId });    await context.Publish(new OrderAccepted { OrderId = message.OrderId });  }}

When everything is working, this handler is fine. A ProcessOrder message comes in, and three messages are produced: a BillOrder command, a CreateShippingLabel command, and an OrderAccepted event.

But what happens if one of those messages fails to be sent? What if BillOrder and CreateShippingLabel are sent, but something goes wrong, and the OrderAccepted event cannot be published? This can be caused by anything from a missing event topic to a momentary network glitch.

If left unchecked, this situation will result in duplicate BillOrder and CreateShippingLabel messages being sent each time the ProcessOrder handler is retried.

We definitely do not want to bill the customer multiple times nor ship them multiple orders. What we want is for the entire operation to succeed or fail atomically. Either all three outgoing messages are produced, or none of them are. If they are produced, then the incoming message should be marked as complete.

Getting this right is not always easy. You need to make sure the incoming message and all of the outgoing messages use the same transaction, and you need to ensure that the message will not be auto-completed by the incoming Service Bus Trigger. Getting it wrong can lead to some subtle bugs that are difficult to detect in a production environment.

So, we made it easy to get it right with just one line of code. If you want to enable transactional consistency between incoming and outgoing messages in your Azure Function NServiceBus endpoint, just tell us, and we’ll take care of the rest:

[assembly: NServiceBusTriggerFunction("MyEndpoint", SendsAtomicWithReceive = true)]

This enables the sends atomic with receive transport transaction mode for ServiceBus and integrates it correctly with the Azure Functions host. You can read more about this feature in our documentation.

🔗IConfiguration

By embracing the Microsoft IConfguration API, we’ve made the setup of your Azure Functions endpoint even simpler.

If you have used any of the new hosting models from Microsoft, you probably are familiar with the new IConfiguration API. This interface allows you to load configuration from files, environment variables, and other sources. This interface is available in Azure Functions as well, as shown here:

public class Startup : FunctionsStartup{    public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)    {        FunctionsHostBuilderContext context = builder.GetContext();        var jsonConfig = Path.Combine(context.ApplicationRootPath, "appsettings.json");        builder.ConfigurationBuilder            .AddJsonFile(jsonConfig, optional: true, reloadOnChange: false)            .AddEnvironmentVariables();    }}

The new release of NServiceBus Azure Functions embraces this configuration interface, so you can use that to configure your endpoints. This is a lot simpler:

public override void Configure(IFunctionsHostBuilder builder){  var configuration = builder.GetContext().Configuration;  builder.UseNServiceBus(() =>     new ServiceBusTriggeredEndpointConfiguration("NServiceBusFunctionEndpoint", configuration));}

This will automatically look up configuration settings such as connections strings and license info directly from the configured sources. In fact, if you add a configuration variable called ENDPOINT_NAME, you can shrink your endpoint configuration to just this:

public override void Configure(IFunctionsHostBuilder builder) =>    builder.UseNServiceBus();

Of course, if you don’t want to use IConfiguration to configure the endpoint or would rather do things the old way, the manual configuration overloads still exist:

public override void Configure(IFunctionsHostBuilder builder){    var endpointConfig = new ServiceBusTriggeredEndpointConfiguration("NServiceBusFunctionEndpoint");    var transport = endpointConfig.Transport;    transport.ConnectionString("MyConnectionString");    builder.UseNServiceBus(cfg => endpointConfig);}

🔗Summary

You hate boilerplate, and so do we. So with the power of C# source generators in our newest version of Azure Functions, you can shrink your endpoint configuration from a couple dozen lines of code (or more!) to a couple assembly-level attributes and a tiny Startup class containing the bare essentials:

[assembly: FunctionsStartup(typeof(Startup))][assembly: NServiceBusTriggerFunction("MyEndpoint", SendsAtomicWithReceive = true)]class Startup : FunctionsStartup{    public override void Configure(IFunctionsHostBuilder builder) =>        builder.UseNServiceBus();}

If only we could do that for all your code.

To check out NServiceBus on Azure Functions, check out our sample, Using NServiceBus in Azure Functions with Service Bus triggers.