Every time a credit card is swiped, tapped, or dipped, a correlation problem has to be solved in real time. Payment systems need low latency and high throughput, but they also need correct outcomes.

For many distributed systems, the common approach is still fairly simple.

A request is made, a response is returned, and the caller moves on. But systems that care about latency and throughput cannot afford to serialize all work behind a single in-flight request on one connection.

Recently, while reviewing a pull request, a discussion arose about using sync/atomic versus sync.RWMutex.

It’s a question that comes up often when writing concurrent Go code, so I thought it would make a great post.

TL;DR

For those in a hurry, here are the key takeaways:

• Atomic Operations: Can be faster when updating a single shared value across multiple threads (or goroutines).
• Mutexes/RWMutexes: Add overhead, but are better for operations involving multiple steps or fields.
• Atomic Operations protect one instruction: They do not guarantee correctness across a sequence of operations (read → logic → write).
• Complexity grows: As logic grows (compare-and-swap (CAS) loops, retries, multi-field consistency), atomic operations become harder to reason about.
• Mental model:
  • Mutex/RWMutex: Everything inside the lock is one unit.
  • Atomic Operations: Only this one operation is protected.
• Rule of thumb: Start with a mutex (RWMutex if read-heavy); switch to atomic operations as an optimization when performance requirements and use cases justify it.

Introduction

In this article, we’ll explore atomic operations and mutexes, how they differ in both performance and behavior, and when each one is appropriate.