Managing APIs is hard. An application usually supports a single version of itself. It can be refactored, restructured, and redesigned with relative freedom. On the other hand, an API has to maintain stability for all its versions that are being consumed.

As requirements change, more and more time is spent on how to avoid breaking changes instead of actually delivering value.

Frozen in Time

An API is a contract between the provider and the consumer. This means that once a version is released, it should remain frozen in time. This is not only true for the contract of the API but also for its implementation. However, this is not a “should” but what happens in practice.

When an API is released, we don’t care about its internal implementation anymore. Yes, there will be bugs and yes, there are multiple shared components. But other than these, we do not make changes. If it works, don’t change it. Right?

Theoretically, we could develop a completely new application for each new version. This would allow us to build something and just make sure we keep it alive. Practically, the cost of maintaining such an approach is prohibitive. Furthermore, this is not how software is built in practice.

Software is built in small incremental batches. So why don’t we optimize our APIs for small incremental changes?

Problem

Code frozen in time sounds good, but what about code that reflects data? Data is also evolving with each new requirement. Take this for example:

 if ($course->active) {
    // do something
 } else {
    // do something else
 }

If we delete the active field in favor of a new status field, then we will have code that depends on a field that does not exist anymore. Even a new application wouldn't help us here as there is still the dependency on the data.

There are ways to mitigate this problem. The easiest one is to check the version of the API and adapt the code accordingly.

 if ($apiVersion >= 2) {
     if ($course->status === 'active') {
         // do something
     } else {
         // do something else
     }
 } else {
     if ($course->active) {
         // do something
     } else {
         // do something else
     }
 }

As you can see, this approach quickly becomes unmanageable. Each new version adds more complexity to the code.

A more common approach is to use feature flags or to split each version in a different folder / class:

 - api
      - v1
          - CourseController.php
      - v2
          - CourseController.php

While this can work, it does not scale well with small incremental changes. Furthermore, you tend to lose what is the latest state of the application.

APIs as Infrastructure

To solve these problems, we decided to treat APIs as infrastructure. An approach first introduced by Stripe back in 2017.

The idea is simple. Your code always reflects the latest version of your API. Each time you need to introduce a change, you update your code to reflect the new requirements. Then, you add a VersionChange that lets you go back in time.

Instead of branching the system into multiple versions, we move the system forward and let transformations pull older versions backward. This keeps change concentrated in one place instead of fragmented across versions.

Let us build on our previous example. We have a CourseEntity that had an active field in V1 and in V2 introduced a status field. We would update our code to reflect the latest version, which means that our code would use only the status field.

To make sure we won’t break the previous versions we add a VersionChange that restores the active field from the status field.

 class ConvertCourseStatusToActiveChange {
    private string $description = 'The active field was replaced by the status field for ...';
 
    public function apply(CourseEntity $course): CourseEntity {
        $course->active = $course->status === 'active';
        return $course;
    }
 }

When a request comes in, we check the requested version and apply all the necessary changes to bring the data to the requested version.

Here’s why we like this approach:

• The code always reflects the latest version of the API.
• Small incremental changes are easy to implement.
• Each version change has a mandatory description that explains why the change was necessary.
• We can freeze old versions without duplicating code.

Treating APIs as infrastructure lets us evolve safely, incrementally, and without fear of breaking the past.

Keeping Versions Aligned With Reality

Most API versioning schemes assume that products evolve through major releases. Versions like example.com/api/v1/courses and example.com/api/v2/courses work well when changes arrive in large batches.

The problem is that major releases require coordination across departments and strict lifecycle planning. More importantly, they contradict everything we have said so far: $small_incremental_changes !== $major_release.

Small, steady changes are easier for consumers to adopt. Ideally, the versioning scheme should reflect that and communicate something meaningful to them.

Date based versioning ( YYYY-MM-DD ) does exactly that. Each version corresponds to a real point in time, making the incremental nature of our changes visible and predictable. It aligns the version history with how the API actually evolves, instead of forcing artificial release boundaries.

Design First, Change Maybe

Code will always remain a technical debt. Having a framework that supports change does not mean that we can avoid thinking about design. Some changes will always ripple through the system. We try to balance between over-engineering and pragmatism.

What matters is creating an environment where change is expected, guided, and safe. A structure that lets us introduce new behavior incrementally, without rewriting the past. An approach where old versions can be frozen with confidence, and new versions can evolve without fear.

By treating APIs as long-lived infrastructure rather than short-lived features, we make this balance possible. We keep the codebase aligned with the current truth of the system, we document why each version exists, and we ensure that past behavior stays accessible without forcing duplication or hacks.

Bonus

While we were experimenting with this approach, we found an open-source project that implements in FastAPI what Stripe describes in their blog post. Having a concrete implementation really helped us to implement this approach in PHP.

You can check it out here: cadwyn .

Originally published at https://blog.talentlms.io on November 17, 2025.

There’s a famous behavioral experiment involving five monkeys, a ladder, and a bunch of bananas.

The bananas are placed at the top of the ladder. Whenever a monkey climbs up to get them, all five monkeys are sprayed with cold water. Over time, they learn: no one climbs the ladder. If one monkey tries, the others pull it down just to avoid punishment.

Then, one monkey is replaced. The new monkey, unaware of the rule, goes for the bananas. The others beat it down. One by one, each monkey is replaced. Eventually, all five are new. None of them have ever been sprayed.

And yet, no one climbs the ladder. If one tries, the others attack.

No one knows why.
It’s just how things are done.

How This Happens in Engineering

This happens in software teams all the time.

We inherit workarounds, rituals, and silent rules:

• Never deploy after 3 p.m.
• Don’t touch that config file.
• The deploy process requires these manual steps.

Over time, these aren’t seen as weird. They’re just how the system works. And if a new engineer asks “Why?” we give them a shortcut or a script. Not an answer.

We’ve stopped climbing the ladder.
And most of us don’t even remember the cold water.

The longer you’re in a system, the more its flaws feel like furniture

The Veteran’s Blind Spot

Engineers who’ve been with a system for years have deep knowledge and deep blind spots.

They know which parts of the codebase are dangerous. They have survived outages, refactors, rewrites. But that survival often makes them too comfortable with dysfunction:

• They stop noticing flaky tests because they’re used to rerunning them.
• They stop fighting tech debt because they’ve already adapted to it.
• They stop advocating for fixes because they’ve internalized the pain.

Eventually, they start teaching the next generation how to survive and not how to improve.

From Broken to Policy

There’s a moment when a system’s flaws stop being technical and they become cultural.

We stop saying, “This is broken.”
We start saying, “This is how we do things.”

Examples you’ve probably seen:

• Broken deploy process? Add a checklist and call it a protocol.
• Legacy API? Add five more client-side hacks.
• Dangerous shell script? Rename it run_this_first.sh and pin it to Slack.

We start to write documentation that doesn’t fix pain, it just translates it.

When broken becomes normal, you’ve stopped evolving.

Why We Accept the Pain

We don’t tolerate this stuff because we’re lazy. We do it because it feels safer than change.

• Fear: “If we touch it, we might break everything.”
• Fatigue: “We’ve tried fixing it. It’s not worth the fight.”
• Familiarity: “We know how to work around it.”
• Bias: “If it’s survived this long, it can’t be that bad.”

These are natural reactions. But when everyone in the team has them, nothing changes. We just pass the pain downstream, from engineer to engineer.

Signs You’ve Inherited Dysfunction

Think your team might be stuck in monkey-mode? Look for these:

✅ Pain that everyone works around, but no one logs
✅ Tribal knowledge passed through Slack, not docs
✅ Engineers who solve symptoms, not causes
✅ Retros with no mention of chronic friction
✅ “That’s just how it is” used to end discussions

If your team feels “comfortable”, ask: comfortable with what?

How to Break the Pattern

You don’t need a heroic rewrite.
You just need people who are willing to ask “Why?”

🔁 Rotate Ownership

Let fresh eyes tackle old systems. New team members see what veterans overlook.

🛠 Budget for Cleanup

Spend 10–20% of each sprint addressing long-ignored pain. Convert tech debt to team debt.

🧭 Reward Curious Engineers

Celebrate people who question process. Normalize dissent. Promote refactoring, not just patching.

🧼 Make Pain Visible

Write honest documentation. Label things as fragile, flaky, or painful. Let visibility lead to action.

Rethinking the Ritual

Culture is what we normalize.

That’s why this article isn’t about broken code, it’s about broken behavior. The longer we let dysfunction become invisible, the harder it is to see the opportunities for change.

The monkeys in the experiment weren’t irrational.
They were just following precedent.

Engineers aren’t irrational either.
But if we never ask “Why?”, we’ll keep beating down the ones who try to fix things.

You don’t have to be the engineer who built the ladder.

But you can be the one who climbs it again.

A platform team’s job is not to enforce standards, it’s to reduce friction.

Team Topologies introduced a powerful framework for structuring engineering organizations around flow, clarity, and responsibility. Among its four team types, platform teams are designed to support stream-aligned teams, reducing their cognitive load by providing reusable, internal services and tooling.

But if platform teams are all about enablement, how can they start slowing everyone down?

Thinking the Platform Is the Product

When platform teams grow distant from the developers they serve, they begin to act like there are no customers at all. This illusion fuels isolation, prescriptive thinking and misalignment. The platform becomes detached from the actual needs of its users . Ironically, they are just down the hall (or Slack channel).

Instead of accelerating delivery, the platform becomes another source of drag.

Common symptoms include:

• Shipping features based on roadmap fantasies, not real requests
• Focusing on code, not usability or feedback
• Measuring success by delivery, not adoption

If nobody uses it, it doesn’t matter that it’s beautiful.

✅ Fix it: Treat your work like a product. Talk to users. Validate ideas. Measure satisfaction. Iterate.

Operating in the Shadows

Ideally, enablement should happen frictionlessly. But without proper communication, it’s just impossible. However, some have cracked the code in a funny way. They have completely removed themselves from the process.

They vanish in isolation and avoid any ( proper ) communication. Then, on the day of “the big release”, something magical happens. A Slack message!

- Hello team, the new API that we have never asked on how it should work or be integrated with your code is finally here. You can use it by following…

- Wait.. who are these people again?

The result?

• Surprise rollouts that break workflows.
• Features teams don’t want or understand.
• No feedback loops.

Internal Marketing = Alignment
An invisible team isn’t a humble team. It’s a disconnected one.

✅ Fix it: Build in the open. Share your roadmap. Market your features internally. Visibility builds trust.

Shifting from Enablers to Enforcers

Being in a position where others request for help is a powerful, yet dangerous, one. What was once just a structure for flow, has become a structure for control. Often, the drift doesn’t happen due to malice but due to lack of communication and teaching skills. It is easier to tell someone what do to than to make them understand why.

While their intention is consistency, the result is resistance.

Stream-aligned teams start:

• Building shadow tools.
• Delaying adoption.
• Avoiding the platform team entirely.

Mandates kill trust. Enablement builds it.

✅ Fix it: Offer opinionated defaults, not hard rules. If your platform is truly helpful, teams will want to use it.

Solving for Hypotheticals, Not Humans

Too many platform teams try to future-proof everything:

• Designing for edge cases that may never come.
• Building complex frameworks “just in case.”
• Abstracting problems no one actually has.

Meanwhile, their customers struggle with basic pain points: slow CI, broken deployments, confusing APIs.

Elegance is worthless if it ignores reality.

✅ Fix it: Walk in your customers’ shoes. Sit in their standups. Observe their pain. Then build what they need, not what looks good on a whiteboard.

Measuring Output, Not Outcomes

If you’re measuring success by:

• Number of microservices created,
• APIs published,
• or internal uptime alone…

You’re missing the point.

Platform teams should care about:

• Time to onboard a new service.
• Number of support questions.
• Developer satisfaction.
• Feature adoption rate.

✅ Fix it: Instrument everything. Not just logs, but feedback. Track usage, not just deployment.

No Clear Interaction Model

Team Topologies stresses the importance of interaction modes: X-as-a-Service, collaboration, facilitation. Without them, platform teams default to either building in isolation or pairing endlessly without strategy.

When teams don’t know how to engage with you, they usually won’t.

✅ Fix it: Make engagement easy and explicit:

• Offer onboarding sessions.
• Run workshops.
• Publish “How to work with us” guides.

The easier you are to work with, the more valuable you become.

You’re Not Building a Platform. You’re Building Flow

At the center of the model are stream-aligned teams. Everyone else, including platform teams, exists to support them as enablers. That means your success is measured by their success.

A great platform team enables stream-aligned teams to:

• Deliver faster,
• Struggle less,
• And spend more time on product.

To get there, you must:

• Treat developers as your customers.
• Stay close to their reality.
• Deliver value early and often.
• Create feedback loops.
• And never forget: you don’t exist to control. You exist to empower.

Update: The rule was merged into PHPStan itself! You can find the PR here: phpstan/phpstan-src#3294

During a refactoring in one of the projects that I work on, we threw away a few legacy classes that where mostly unused. Since the project is a legacy one, the "autoloader" for the classes was actually a require_once . Unfortunately for us, we missed a few of these statements in some of our scripts and they crashed the application when being called. The reason is that require_once throws a fatal error if the file does not exist.

Besides the human factor, what went wrong and how do we prevent this from happening again?

We had all these fancy tools on our toolchain ( PHPStan, CodeSniffer ) so how did we ended up with such a trivial bug?

Well, it turns out that neither of them actually evaluates the arguments passed to the require , include statements 😱

Let there be a custom rule: PHPStan 1:3

So, we found why we didn’t catch it ( it never worked dah ) let’s find a way so it never happens again.

You can skip the article and dive right into the code: https://github.com/Bellangelo/phpstan-require-file-exists

The best candidate for such a thing seems to be a custom rule in PHPStan. To make PHPStan understand our rule we just need to implement its interface:

use PHPStan\Rules\Rule;

class RequireFileExistsRule implements Rule {}

After this, we need to specify what type our Rule listens to through the getNoteType() method. Fortunately for us, PHPStan and PhpParser have done an amazing work to unify several node types of PHP under a single node. In our case, require , require_once , include , include_once are all under the Include_ class.

Here how our code transforms:

use PHPStan\Rules\Rule;
use PhpParser\Node\Expr\Include_;

class RequireFileExistsRule implements Rule {

  public function getNodeType(): string
  {
    return Include_::class;
  }

}

Now, the “only” missing part is to accept the nodes we requested. This happens through the processNode(Node $node, Scope $scope) :

public function processNode(Node $node, Scope $scope): array
{
  if ($node instanceof Include_) {
    $filePath = $this->resolveFilePath($node->expr, $scope);
    if ($filePath !== null && !file_exists($filePath)) {
      return [
        RuleErrorBuilder::message(
          sprintf(
            'Included or required file "%s" does not exist.',
            $filePath
          )
        )->build(),
      ];
     }
    }

    return [];
  }

Let’s go line-by-line to see what it does.

1. First, we check that the passed $node is an instance of Include_ .
2. Then, we resolve the file path ( through magic for now ) by passing the nodes of the Include_
3. If the $filePath is not null and the file does not exist then we return an error. We want to make the $filePath be null in case we cannot evaluate it.

Of course, inside the method all the magic happens in the resolveFilePath . We need to find out what nodes can a Include_ contain and handle every possible case. It turns out, that almost every possible node can exist inside a Include_

String_

The easiest node to start with, is of course, a string. Thankfully, it doesn’t require any special handling:

if ($node instanceof String_) {
   return $node->value;
}

Dir

The most common way of creating absolute paths, __DIR__ . Once again, PhpParser is on our side and has a node for it. The only catch is that it doesn’t provide any value that we can use to calculate the filename. Instead, we need to use the Scope that PHPUnit provides us with:

if ($node instanceof Dir) {
   return dirname($scope->getFile());
}

Concat

An enexpected but real thing. When you use concatenation most static analysis tools convert it into a different node type. This node type can then contain almost anything. Sounds like a job for recursion:

if ($node instanceof Concat) {
   $left = $this->resolveFilePath($node->left, $scope);
   $right = $this->resolveFilePath($node->right, $scope);
   
   if ($left !== null && $right !== null) {
      return $left . $right;
   }
}

Can we do better?

Obviously, it is a lot more difficult to handle variables, method, functions etc etc. Anything that is more dynamic than our previous cases. Although, class constants might be a good candidate. Let’s try to check if the Node is a class constant and fetch its value through reflection:

if ($node instanceof ClassConstFetch) {
 return $this->resolveClassConstant($node);
}

private function resolveClassConstant(ClassConstFetch $node): ?string
{
  if ($node->class instanceof Node\Name && $node->name instanceof Node\Identifier) {
    $className = (string) $node->class;
    $constantName = $node->name->toString();

    if ($this->reflectionProvider->hasClass($className)) {
      $classReflection = $this->reflectionProvider->getClass($className);

      if ($classReflection->hasConstant($constantName)) {
         $constantReflection = $classReflection->getConstant($constantName);
         $constantValue = $constantReflection->getValue();
         
         if (is_string($constantValue)) {
           return $constantValue;
         }
      }
    }
  }
  
  return null;
}

There should be other cases as well that we can do a better job, such as global constants ( define ) but you are more than welcome to open a PR for any case that thought is missing: https://github.com/Bellangelo/phpstan-require-file-exists

Don’t forget to star the project if you like it ⭐️

You can always install it by running:

composer require bellangelo/phpstan-require-file-exists

In recent years, the Microservice architecture still has attached some “coolness”. While the term premiered 13 years ago, the promises that it brings still holds up the hype around it. However, as most solutions, Microservices are often adopted for the wrong reasons or to put it better, for the problems that they do not have ( yet ). Is the hype justified, or are we overlooking simpler, more effective solutions?

The promise of the Microservices

Microservices are all about scaling, whether is code or people. This architecture lets you manage and deploy certain “aspects” of your application independently from the others. Thus, converting your applications into a collections of mini / micro services. The new size of each new service forces a reduction to its scope itself. Sometimes, this is the goal itself. A reduced scope comes with increased manageability. But by who? Of course, by people. For good or worse, the abilities we value most ( readability, reusability, etc ), ultimately depend on the people who interact with the code.

Local complexity vs Global complexity

The run to optimize each new service is not a simple marathon but a triathlon. To name a few of the main challenges:

1. Inter-service communications: What do you mean the network is down? Should I retry the request? Wait.. did I just DDOS myself?
2. Transactions: Hmm.. for some reason transactions stopped working. What did you say? Saga pattern, orchestration? Stop talking nonsense, we are engineers, not musicians.
3. Dependency management / Deployment: Finally, every service can be deployed independently. What do you mean that the Service C needs to be deployed with a specific version of Service D?

Not all of these challenges can be seen when you look at just one service. On our local little island, we don’t care about outsiders. Although, we should. The Microservices architecture splits the application by bringing another player to the game, the network. Although appealing, reliable communication between services is challenging due to potential network failures and message delays.

Part of the network problem is transactions. Again, transactions inside our local island will work perfectly fine. The problem are the distributed transactions. Managing transactions across distributed services often requires complex patterns like Sagas and Orchestration, which can add significant overhead and complexity.

Last but not least, we have independent deployment. While it is a key benefit of microservices, but inter-service dependencies can complicate the process, sometimes necessitating synchronized deployments.

Microservices bring many things and one of them is complexity. We all want our service to be fully optimized, but we cannot do it without considering the complexity of the whole system.

Introducing Modular Monoliths

Modular Monoliths take a local perspective of the Microservices’s promise.

Network

Since most of the things happen locally, we can immediately remove the network. Without the network, the problem of the inter-service communications disappears, or depending on the application it is reduced.
So far, this sounds exactly like a normal monolith. The difference is that we will be keeping the services.

Modules

A Modular Monolith is split into Modules. Each Module should mostly work as a service in a Microservice. This means:

1. Encapsulation: Each module encapsulates its own functionality, keeping its internal implementation hidden from other modules. This ensures that changes within a module do not affect others, promoting stability and ease of maintenance.
2. Defined Interfaces: Modules communicate with each other through well-defined interfaces or APIs. This clear separation of concerns allows for more organized and manageable code, making it easier to understand and develop.
3. Independent Development: Teams can work on different modules independently, similar to how they would handle Microservices. This parallel development can speed up the overall development process and reduce bottlenecks.

Dependency management / Deployment

While Modules operate like a service they still co-exist in the same codebase. This means we can have shared resources between and most importantly, handle them as a single deployment unit. Thus, eradicating completely the need for a complex deployment system.

No silver bullet

Modular Monoliths sounds really great when compared to Microservices. The truth is that there is no silver bullet when we are talking about architecture. The only thing that separates a Module from another Module is the folder structure. Everything else is shared, creating a resource contention.
If everything is shared then so is scaling. Unlike Microservices, which can be scaled independently based on their specific demands, a modular monolith requires scaling the entire application.
Bypassing the filesystem is something that everyone can do. Even with modularization, there is a risk of modules becoming tightly coupled over time. This can happen due to shared code, common libraries, or direct references, making it harder to maintain a clean separation of concerns.

Conclusion

Choosing the right software architecture is crucial for the success of any project. The introduced complexity by the Microservices does not always justify the remarkable scalability and flexibility it offers. Conversely, Modular Monoliths offer modularity while keeping a simple single deployment unit , yet they still face challenges such as resource contention and scaling limitations.

The key to selecting the best architecture lies in understanding the specific context and needs of your project. Factors such as team size, expertise, project scale, and long-term maintenance when making your decision. Sometimes, the simplicity and efficiency of a modular monolith can be the perfect fit for a project’s requirements, while other times, the granularity and scalability of Microservices are necessary to meet the demands of a growing application.

If you are starting a new project, choosing a Microservice approach will be like shooting yourself in the foot. Most projects require just a Monolith and an early stage project is one of them. Ultimately, the most important aspect is to make an informed choice that best supports your team’s capabilities and your project’s success.

Efficiently managing database interactions is crucial for performance and scalability. A common element in database interactions is the opening and closing of connections. Could there be a way to streamline this process by sharing connections across multiple requests?

Basics of Database Connection Pooling

What is Database Connection Pooling?

Database connection pooling refers to the practice of maintaining a cache of database connection objects. This cache, or “pool,” allows for the reuse of connections, eliminating the overhead of establishing a new connection with each database interaction.

How It Works

When an application needs to execute a database query, it retrieves a connection from the pool rather than opening a new one. After the operation, the connection is returned to the pool for future use. This process significantly reduces the time and resources spent on opening and closing connections.

Advantages of Connection Pooling

Performance Improvement

By reusing existing connections, connection pooling minimizes the latency associated with establishing new connections, leading to faster query execution.

Resource Management

Connection pools help in efficient resource management by controlling the number of active connections, thus preventing database overload.

Scalability

With connection pooling, applications can handle more concurrent database operations, enhancing scalability.

Implementing Connection Pooling

Choosing a Pooling Library

Selecting an appropriate connection pooling library is pivotal. Each library has its unique features and configurations.

Configuration Parameters Key parameters to configure in a connection pool include:

• Maximum and Minimum Pool Size: Determines the pool’s capacity.
• Connection Timeout: The maximum time to wait for a connection from the pool.
• Idle Connection Test Period: Frequency of checking idle connections.

Best Practices

• Monitor Your Pool: Regular monitoring can help in fine-tuning the pool size and parameters.
• Exception Handling: Implement robust exception handling to deal with scenarios where connections are unavailable.
• Resource Cleanup: Ensure that connections are properly closed and returned to the pool.

Tailoring Pooling Strategies to Language Environments

Shared Process Languages

In programming environments where multiple requests are handled within the same process, such as in Java or .NET applications, the implementation of connection pooling can be directly integrated within the application. This is because these environments typically run a continuous process that can maintain state (like a connection pool) across multiple requests.

How It Works in Shared Process Languages

• Shared Memory Space: Since all requests share the same memory space, a connection pool can be maintained as a global or singleton resource accessible by all parts of the application.
• Long-Lived Processes: These environments usually have long-lived processes (like a web server or application server), which makes it feasible to keep a pool of connections alive and readily available for incoming requests.

Benefits in Shared Process Environments

• Immediate Access: Connections are immediately available to any request processed by the application, reducing the overhead of connection creation and disposal.
• Resource Efficiency: This approach maximizes the efficient use of database connections, as the pool is centrally managed and optimized by the application itself.

Implementation Considerations

• Thread Safety: Ensure that the connection pool is thread-safe, as it will be accessed by multiple concurrent requests.
• Configuration Tuning: Optimize the pool size and other parameters based on the application’s load and database usage patterns.
• Monitoring and Management: Continuously monitor the pool’s performance and health. Implement management features like connection validation and aging to maintain pool efficiency.

Process-Per-Request Languages

In languages where each request runs in its own process (such as PHP or Python), an external service is needed to manage the pooling. These services vary based on the database system in use. For PostgreSQL, PgBouncer is a widely used and actively maintained solution. For MySQL and its derivatives like MariaDB, ProxySQL is a highly efficient choice.

Conclusion

Database connection pooling is an essential strategy for enhancing the performance and scalability of applications. By understanding its principles and implementing it effectively, software engineers can achieve significant improvements in database interaction efficiency. Embracing these practices not only optimizes resource usage but also paves the way for more robust and scalable application architectures.

Database replication is a common practice for scaling your application. It involves deploying multiple copies of the same database to ensure data availability and reliability. While this horizontal scaling is simple in theory, it still hides a lot of complexity. Since a request can hit a different replica each time it can lead to inconsistencies and confusion.
If you developed a system like this, congrats. You just, accidentally, made a time machine!

Understanding Database Replication

Before diving even further on the “time machine” issue, let’s better understand what database replication is. At its core, database replication is the process of copying and maintaining database objects, like tables, in multiple location. This can happen in real-time or in near-real-time.

The operation can be:
Synchronous: This means that for the operation to be considered completed, the changes must be written to all the replicas.

Asynchronous: The changes are synchronised at a later time allowing potential temporal disparities.

Why Replicate?

Replication serves several key purposes:

1. High Availability: By having multiple copies, we can ensure that the data remains accessible. In the event of the hardware failure, network issue, or other disruptions another replica can serve the request.
2. Load Balancing: Multiple workers mean less work for each worker. Distributing the load across multiple servers reduces the load that reaches each server and as such, enhance the performance.
3. Disaster Recovery: In catastrophic events, having data replicated in multiple geographic locations can safeguard against data loss.

However, replication comes with some challenges. The foremost among these is ensuring consistency across replicas. Systems that favor high availability and partition tolerance are the most common targets that are “hit” by this challenge.

The Time Machine Issue

In layman’s terms, the time machine issue emerges when different replicas contain data from different moments in time. This situation can lead to paradoxical scenarios where multiple identical requests can result in different responses, despite the requests and the underlying queries being simultaneous. The root of this problem lies in replication lag — the time it takes for changes made to the primary database to propagate to its replicas.

Impact on Users

For users, this inconsistency can be baffling and frustrating. Imagine an e-commerce scenario where a customer clicks to view a product from the search but it results to a “404 — Not Found” page. Such experiences reduce the trust and reliability in the system.

Tackling the Challenge

Addressing the time machine issue requires a many-sided approach. Here are some strategies:

1. Improved Synchronization: Optimising the synchronization protocols can minimise the replication lag and ensure the replicas are as up-to-date as possible.
2. Read-Write Splitting with Smart Routing: Directing all write operations to a primary node and reads to replicas, coupled with intelligent routing that considers data freshness, can mitigate the risks of reading stale data. The database itself might even be able to handle this by marking which data are “ready” to be served. This combines an asynchronous approach with a synchronous mechanism.
3. User-Centric Solutions: I am sure you have heard that session stickiness is a bad thing, but what about database connection stickiness? Connection stickiness can ensure that a user will hit the same replica in all of his requests. A simple implementation to this consists of creating a function that returns a replica based on the user’s id. Depending on the application’s needs even this might not be enough. For example, if you have a SaaS product and customer has its own portal you might want your function to be based on the portal’s id. This way, you achieve consistency among all the users you “care” about. An even more complex scenario might be a social network, where each user is everywhere and deciding for a specific replica is well.. complicated.

Conclusion

In conclusion, the “time machine” issue in database replication is a complex problem that requires careful consideration and tailored solutions. By understanding the root of the problem and employing strategic measures based on your business needs, you can ensure reliable and accurate data to users.

A well designed service in software architecture must be modular and have a single responsibility. Yet, as services evolve and mature, the demand for new features is increased. While some of them can be developed as a whole new service there are some cases where a more integrated approach is preferred. In this article, we will discuss how the sidecar pattern can be used to extend our services while keeping its resiliency.

What is the Sidecar Pattern?

The Sidecar pattern is nothing more than an additional container. This container runs alongside your main application’s container. Unlike traditional practices, the sidecar container shares resources with the application container such as file system and network. This is done so it can seamlessly enhance the application’s functionality. Most often, this happens without the knowledge of the application. Although, the application might not know about the sidecar container their lifecycle is mutual. This means that they are deployed and handled together as a single application.

Benefits

1. Isolation of Responsibilities: The sidecar pattern can keep supportive functionalities outside from the main application. This means that features such as logging, monitoring and configuration can be handled by other teams ( DevOps ).
2. Improved Scalability and Performance: By offloading certain aspects of the application to sidecar components, the main service can perform its core functions more efficiently. By having a “close friend” to handle the heavy load the application itself can keep up with the demand without any disruption.
3. Enhanced Security and Reliability: In the sidecar pattern, separate components handle critical security tasks such as authentication and encryption. This approach adds a robust layer of security without cluttering the primary service with complex security protocols. Moreover, it boosts system reliability by localizing and managing fault handling and recovery processes within these sidecars.
4. Easier Upgrades and Maintenance: This is a side-effect of the isolation of responsibilities. As each container is isolated it means that it can have different maintenance and upgrade cycles.
5. Flexibility in Technology Choices: Running a new container abstracts us from the “problems” of the main service. This gives us the flexibility to choose the right language and tools for the right job.

When not to use it?

1. Simple or Monolithic Applications: The Sidecar pattern is used to reduce complexity. For simple or truly monolithic applications, this pattern might introduce unnecessary complexity.
2. Tight Coupling Needs: If the application requires tight coupling between its main functionality and supportive services (like logging or monitoring), using the sidecar pattern can be counterproductive. The pattern works best when there’s a clear distinction and independence between the main service and its sidecars.
3. Limited Resources or Infrastructure: Running 2 containers as a single application requires orchestration. Having a limited infrastructure environment that doesn’t provide such options make this pattern difficult to implement. Furthermore, containers requires more resources than bare-metal which is another thing to consider.
4. Performance-Sensitive Applications: If performance is a critical factor, an esoteric approach might be preferred. Sometimes, even the small extra hops that the data need to do over the local network might be not viable.
5. Small Development Teams: An additional container is an additional thing to maintain. Small teams usually work better when they manage a monolithic application and the sidecar pattern is quite the opposite.\

Strategic advantage in software development

The Sidecar pattern offers a significant strategic advantage in software development, particularly in the context of collaboration with DevOps teams. By adopting this pattern, developers can utilise plug-n-play modules designed by a specialised team. Solution such as logging and monitoring can be standardised and re-used across different services and teams. This not enables new services to be production-ready immediately but also retracts the focus of the developers to the main application.

Real world examples

Having explored the Sidecar pattern in theory, let’s examine its real-world applications.

Adding HTTPS to a legacy service

Not all services are designed to accept HTTPS, especially in the legacy universe. Due to how coupled a legacy service finding a solution that does not require any changes to the main service is always a good idea.
Imagine that your service listens to 127.0.0.1. This means that our service listens only to local connection. We can add a Nginx service through the Sidecar pattern that share the same network as our main service. This way, our Nginx service can listen to outside traffic, in whatever protocol, and pass it to our legacy service. We have successfully modernised our legacy application by developing, basically, a SSL Proxy.

Auto-updating of configuration

There are cases where we need a service to automatically update its configuration from a specific source. This is not usually the focus of the service itself so we can outsource it to an external service. Taking advantage, again, of the Sidecar pattern we can create a container that share the same file system with our application’s container. This way, even a simple git pull loop would be sufficient to fetch any new changes. Depending the case, a more sophisticated approach could be used such as web hooks or by directly providing an API.

Modernising asynchronous jobs

Imagine, that we have a legacy application that runs asynchronous jobs by calling directly a script ( exports, mass actions, etc ). After a while, we want to have more control over the whole process as there are many cases of resource exhaustion. To separate the resources we will utilise the Bulkhead pattern and deploy the scripts and their dependencies in a different environment than the main service. Sounds good in theory, but since the scripts run in a different service now, how can we run them?
Following the modern standards we choose to follow the Publisher — Subscriber pattern for the communication part but still, our legacy scripts cannot be refactored. For this reason, we will deploy the whole new approach as an extension to our scripts. This means, that the Subscriber service will exist in the sidecar container which will share the same file system as the container of our scripts so it can call them. We have successfully transitioned a part of our application to microservices, even though the container for our scripts remains a monolithic application in itself.

Summary

The Sidecar pattern enhances, rather than complicates, the primary service through isolated yet connected components. We saw how it allows for a neat division of labor, entrusting supportive functionalities like logging and monitoring to specialized sidecar components. While all of these sounds good, as we discussed, it is not a pattern that can be used for all the cases. Although, in the right problem it can even improve the progress of your teams.

In summary, our exploration of the Sidecar Pattern reveals it as a harmonious blend of functionality, security, and efficiency, while respecting the principles of microservices architecture.

In the era of microservices, it’s common for services to either depend on others or act as dependencies. These situations call for a method to robustly safeguard our application’s components when a dependency acts up or doesn’t respond at all.

The Bulkhead Pattern

Coming from ship design, where bulkheads divide a ship into sections to stop flooding, this pattern does something similar for microservices, helping to keep them safe. It mainly focuses on the isolation of the resources and thus preventing a whole systemic failure. The isolation is achieved through resource pools consisting of anything that can be exhausted ( CPU, GPU, memory, etc ).

Read world example: Read vs Update

Moving beyond theory, let’s dive into the real world. Supposedly we have 2 endpoints:

• GET /products/{id} : Returns the data of a single product. It is a lightweight action that its response can easily be served directly from the cache.
• PATCH /products/{id} : Updates the data of a single product. This ‘heavy’ action may involve updating data across multiple storage systems and invalidating cache entries.

We can easily see that the PATCH action is a more resource-intensive request that can easily hurt the performance of the application.Consider the scenario where this endpoint receives multiple simultaneous requests. A lot of write requests will hit the database and start clogging the available connections. Of course, the issue is not isolated and it affects also the users that just want to view the product’s data.

A typical solution is to set a maximum number of threads that can handle this endpoint. This will work as a rate-limiter and will minimise the maximum resources that it can consume.

Real world example: Service as a dependency

Goind straight to the point, we once again have 2 endpoints:

• GET /movies : Returns a list of movies. Its response may or may not be served from the cache.
• GET /movies/{id} : Returns the data of a specific movie. To construct the necessary data it has 2 dependencies:
  1. RatingsService: It returns the ratings for the movie.
  2. CommentsService: It returns the comments for the movie.

The first endpoint is straightforward, with minimal dependencies. Caching its results could significantly boost performance.

On the other hand, the movies/{id} is dependant to the performance of its dependencies. While we might have isolated each service’s resource, a reduced performance in any dependency will affect the “parent” service and probably the other dependencies as well. To protect the resources of the “parent” service a rate-limiting approach might be insufficient. The reason for this is that the idling process might be more stress to other services or exhaust the resources of itself. Through the usage of time-outs and fallbacks we can still return a partially correct response and prevent the failure of the whole service. For example, even the RatingsService is down or it has a degraded performance we can still return the movie’s main data and its comments.

We increased the isolation even more but what if a service starts acting up? how do we stop from making things worse?

The Circuit Breaker pattern

In a distributed environment each service is autonomous. By definition, this makes each service unreliable. A server can be on fire or simple network errors might occur. Because of this uncertainty there is a need for a service to alert when it has stopped accepting requests.

It sounds good, but why do we need an external service to know when another service is unresponsive? we can just ping it directly, right?
The answer of every good engineer, it depends. For simple applications this might be sufficient and you might never have a problem. For bigger application the things are a little bit different. When a service becomes unavailable, its requests will start to pill-up and eventually might be too many for the rervice to restart itself and start again. So an infinite process of pinging and restarting might happen.

Here comes to the rescue the Circuit Breaker pattern. We basically build on top of our components another service that is responsible to monitor their health and proxy the traffic to them. In case the perfomance of a component gets degraded the Circuit Breaker stops proxying the traffic to the service. Using a controlled pinging system (where only one client checks health status), the Circuit Breaker can resume directing traffic to the component. More complex criteria can set about when the switch is activated but the main idea stays the same.

Summary

In our adventure of resilient systems we have discussed two pivotal patterns: The Bulkhead and the Circuit Breaker. The Bulkhead pattern, drawing inspiration from naval architecture, emphasizes the importance of resource isolation. It ensures the issues of one segment do not escalate into a systemic failure.

On the other hand, the Circuit Breaker addresses the challenges of the inter-service dependencies in a distributed environment. It keeps an eye on services’s health and carefully manages the flow of requests to stop the system from getting overwhelmed. This pattern is really important for dealing with unexpected problems like network issues or server failures.

In conclusion, both patterns play a critical role in building robust and resilient microservices architectures. By isolating services and smartly managing traffic, these patterns collectively enhance system stability and reliability. Their application can significantly uplift the resilience and efficiency of distributed systems, ensuring smoother and more reliable operations in dynamic and challenging environments.

Most probably you will have both problems.

Domain Events

What Are They?

A domain event is essentially a message that signifies an action within our business context.
Examples: User Registered, User Bought Course, Course Assigned to User.

How to use them:

Domain Events are mostly used with the combination of Event-Driven Architecture. In this architecture, we emit events that subscribers respond to, ensuring a separation of concerns as each subscriber operates independently.
In the context of Domain-Driven Design (DDD), we can link different aggregates to a specific business event. For example, once an aggregate completes a business action, it immediately emits a business event to signal this action. Subscribers, which could be other aggregates or processes, then execute their own business logic and update their states accordingly. It’s important to note that the publishers and subscribers operate independently, without concern for who is listening to the event.

What is the Difference Between a Domain Event and a Regular Event in Event-Driven Architecture?

While both types of events might utilize the same queuing and publishing mechanisms, a key distinction exists. Domain Events are, at their core, business events. Their naming should reflect this, adhering to our business language to ensure comprehensibility among all project stakeholders.
Another notable difference lies in the use of tense in naming. Domain Events should always be in the past tense as they describe actions that have already occurred. In contrast, events in Event-Driven Architecture can be direct commands or updates for state changes. For instance, a ‘User Update Email’ event might prompt an aggregate to change its state.
To summarize, Event-Driven Architecture allows for events to function as commands or notifications about actions, whereas Domain Events specifically indicate that a business action has occurred.

Domain Services

Most of the time, certain logic is shared across different domain contexts, aggregates, or even value objects and typically does not belong exclusively to any of them. Domain Services are the solution to this challenge.
A Domain Service is nothing more than a stateless object that handles the business logic by working as co-ordinator between different parts of our application.
While a Domain Service can handle multiple aggregates we should keep in mind the atomic operation of them. We shouldn’t use them as way to bypass the transactional life of the aggregates. Instead, Domain Services should be seen as a means to execute calculations across different aggregates and facilitate data sharing.

What to use and when?

Deciding whether to use Domain Events or Domain Services hinges on the specific requirements of your application’s functionality. In practice, most applications benefit from the integration of both.

• Use Domain Events when you need to signal that something significant has occurred in your business domain. They are ideal for scenarios where various parts of your application need to react to specific events, but these reactions are independent of each other. For example, after a user registers, you might have multiple systems that need to respond to this event, like sending a welcome email or updating a user database.
• Use Domain Services when there’s a need for complex business logic that spans multiple domain entities or aggregates. They are especially useful for orchestrating processes that involve multiple steps or interactions between different parts of your system. For instance, a Domain Service might coordinate a series of steps to complete a financial transaction, involving validation, updating multiple records, and notifying the relevant parties.

In essence, choose Domain Events for decentralized, reactive processes and Domain Services for centralized, complex business logic coordination. Often, these two concepts will coexist and complement each other within a well-architected application.