Coding

Sponsored By: NutanixThe following is sponsored content. It may not reflect the views of our editorial staff. The Kubernetes scaling problem nobody talks about Enterprise platform teams encounter the same pattern repeatedly: a Kubernetes platform works well enough that nobody wants to change it. This happens gradually as teams make reasonable technology choices: selecting different ingress controllers, secrets management tools, CD platforms, or observability software. Individually, none of these decisions is a problem. Months later, however, they’ve created a Kubernetes environment that only a handful of people understand. As soon as that one person gets sick or leaves the company, maintaining or improving the platform becomes much more difficult. Mark Dastmalchi-Round, a Solutions Architect at Nutanix with decades of experience in platform engineering, describes the pattern in blunt terms: “Configuration drift, exacerbated by the fact that multicloud is increasingly becoming the new reality.” Over time, that drift compounds. Companies get acquired, technology merges, and silos form. Suddenly, organizations are managing clusters that look nothing alike and are often held together by institutional knowledge. As a solution, proprietary overlays have sought to address these issues, with mixed results. They tend to reduce overall surface area (fewer choices lead to fewer points of divergence), but often at a cost to portability and extensibility, which is what made Kubernetes so attractive in the first place. A more durable approach is to build on Kubernetes-native primitives, adding governance and operational consistency without replacing the workflows teams already use. The remainder of this article will demonstrate what that looks like in practice. What an open platform actually means in enterprise Kubernetes “Open platform” is a common phrase in the Kubernetes ecosystem, but it’s worth defining what that term actually means in practice. Dastmalchi-Round defines an open platform as one that “exposes industry-standard APIs and, where possible, uses pure upstream open-source projects.” The distinction isn't whether the platform is open source. It's whether it relies on Kubernetes-native APIs and tooling or introduces proprietary CRDs, workflows, and CLIs that make migration difficult. As he notes, "You can still get lock-in with open source, because if it is only one vendor's solution and they layer all of their stuff on top of standard tooling, you are now dependent on their abstractions." The difference is easier to see when comparing an open platform with a proprietary overlay. Comparing Open Kubernetes Platforms and Proprietary Overlays Dimension Open Platform (NKP) Proprietary Overlay Core CRDs Standard upstream (Cluster API, FluxCD, Helm) Vendor-specific, migration cost is high GitOps engine FluxCD (CNCF project) Proprietary sync engine App packaging Helm + OCI (industry standard) Custom catalog format Monitoring stack Pure upstream CNCF (Prometheus, Grafana) Wrapped / vendor-branded Exit cost Clusters survive platform removal Manifests tied to platform APIs Third-party tooling Works if it runs on Kubernetes Requires certified integration Nutanix Kubernetes Platform (NKP) applies these principles by building on upstream Kubernetes components rather than replacing them. As Dastmalchi-Round puts it, the real test is what survives if you remove the platform. "With NKP, the clusters are pure upstream Kubernetes,” says Dastmalchi-Round. “The monitoring stack is pure upstream CNCF projects. GitOps is provided by FluxCD. Your manifests and charts are standard Helm." In other words, the operational tooling may change, but the underlying applications and deployment artifacts remain portable. Raw manifests to managed artifacts: Helm and OCI packaging in NKP Most enterprise teams start with a collection of Kubernetes YAML manifests that work for a single application or environment. While those manifests are typically stored in version control, they aren't easily reusable across environments, self-service for other teams, or packaged in a way that supports consistent versioning and rollback. Helm addresses those limitations by packaging manifests into versioned, parameterized charts. For existing applications, the process typically starts by converting Kubernetes manifests into a standard Helm chart, either manually or with tools such as Helmify. The result is a familiar Helm project structure built around Chart.yaml, parameterized templates, and a values.yaml file, giving teams a reusable deployment artifact instead of a collection of static manifests. Deployment-specific settings, such as image tags, replica counts, and resource limits, move into a values.yaml file, while the underlying templates remain unchanged. Those deployment-specific settings are defined in the chart's values.yaml file. For example: # values.yaml — the self-service interface for application teams replicaCount: 2 image: repository: registry.example.com/myapp tag: "2.1.0" pullPolicy: IfNotPresent resources: limits: cpu: 500m memory: 256Mi requests: cpu: 250m memory: 128Mi ingress: enabled: true host: myapp.internal.example.com annotations: kubernetes.io/ingress.class: "traefik" serviceAccount: create: true name: "myapp-sa" Versioning makes deployments reproducible across environments while providing a clear history of releases. Teams can promote the same chart through development, staging, and production with confidence, then roll back to a previous version if needed. OCI registries address the next challenge: distributing and versioning those charts. Instead of relying on a separate chart repository, teams can store Helm charts alongside container images as immutable, versioned artifacts. Because chart versions can't be overwritten, deployments are reproducible and easier to audit. The approach also fits existing registry workflows. Organizations using Harbor, Amazon ECR, or similar registries can manage container images and Helm charts in the same place, using the same authentication, access controls, and security policies. For example: # Package the chart locally helm package ./myapp --version 2.3.0 # Authenticate to the OCI registry (same registry as your container images) helm registry login registry.example.com \ --username $REGISTRY_USER \ --password $REGISTRY_PASSWORD # Push is stored as an OCI artifact alongside container images helm push myapp-2.3.0.tgz oci://registry.example.com/charts # Any team can pull without touching the source repo helm pull oci://registry.example.com/charts/myapp --version 2.1.0 # Inspect the chart before deploying helm show values oci://registry.example.com/charts/myapp --version 2.1.0 The goal of packaging is to create a self-service deployment model. Once packaged, Helm charts are registered with the NKP catalog, where they appear alongside built-in platform applications as versioned deployment artifacts. Application teams can deploy them by configuring only the settings that vary between environments, while platform teams focus on maintaining reusable application catalogs instead of manually managing deployments. FluxCD deployments, overrides, and upgrades Once Helm charts are stored in an OCI registry, FluxCD keeps deployed clusters aligned with the desired state defined in Git. It continuously reconciles each cluster against that source of truth, automatically correcting configuration drift. In multi-cluster environments, each cluster follows the same reconciliation process using its own configuration. NKP's FluxCD implementation centers on two resources: HelmRepository, which points to the OCI registry, and HelmRelease, which specifies the chart version, configuration values, and target namespace. # Source: points FluxCD at your OCI chart registry apiVersion: source.toolkit.fluxcd.io/v1beta3 kind: HelmRepository metadata: name: internal-charts namespace: flux-system spec: type: oci url: oci://registry.example.com/charts interval: 5m # poll for new chart versions every 5 minutes # Release: declares desired state for a specific deployment apiVersion: helm.toolkit.fluxcd.io/v2beta3 kind: HelmRelease metadata: name: myapp-production namespace: production spec: interval: 10m chart: spec: chart: myapp version: "2.3.0" sourceRef: kind: HelmRepository name: internal-charts namespace: flux-system values: replicaCount: 3 resources: limits: cpu: 1000m memory: 512Mi ingress: host: myapp.prod.example.com Although teams interact with NKP through its web interface, those actions are ultimately represented as standard Kubernetes resources. Configuration changes become declarative objects that FluxCD reconciles like any other GitOps workflow, making the deployment model transparent and compatible with standard Kubernetes tooling without relying on proprietary deployment workflows. Teams typically promote the same chart version from development to staging and production while applying environment-specific overrides through HelmRelease values rather than modifying the chart itself. Promotion becomes a Git commit instead of a manual deployment, with FluxCD automatically reconciling and applying the change. FluxCD also provides continuous drift detection. If someone manually changes a resource in the cluster, FluxCD restores it to the state defined in Git during the next reconciliation cycle. Rolling back a deployment is simply a Git revert, with Git history providing a complete audit trail of configuration changes. How to integrate third-party tools without losing openness Enterprise platform teams are often asked to integrate tools such as vulnerability scanners, cost management dashboards, and application performance monitoring (APM) platforms. The tools themselves aren't the problem. The problem is managing each one through a separate deployment and maintenance process, increasing operational complexity over time. NKP addresses this by treating third-party software like any other platform application. Whether it's an upstream open-source project or a commercial product distributed as a Helm chart, it follows the same Helm-over-OCI packaging model and is deployed and managed through FluxCD. The outcome is a consistent deployment and lifecycle workflow across both first- and third-party applications. For example, an upstream Helm chart such as Redis can be published to the NKP catalog and managed through the same deployment workflow as a first-party application, avoiding the need for a separate integration process. Because this approach relies on standard Kubernetes resources, Helm charts, Git, and Kubernetes RBAC, those workloads remain portable across platforms. As Dastmalchi-Round summarizes, "If it works on Kubernetes, it will work on NKP." Dastmalchi-Round notes that the biggest integration challenges typically come from tools that rely on rigid deployment models, particularly older operator-based packages that expose little configuration. "A few years ago, there was a trend of people overusing the operator pattern for packaging applications," he says. "Operators have their uses, but when they became the distribution artifact, they often resulted in big, opaque blobs running in your cluster. If they didn't do exactly what you needed, you were out of luck." As more vendors have adopted Helm-based packaging, those limitations have become less common. Examples of Third-Party Tool Integrations in NKP Integration Type Packaging Model Configuration Upgrade Path NKP Catalog Security scanner (e.g., Trivy) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Custom Grafana dashboard Helm chart + ConfigMap Dashboard JSON in Git Chart version update Yes Cost management (e.g., OpenCost) Helm chart via OCI values.yaml in Git FluxCD HelmRelease bump Yes Service mesh (e.g. Istio) Helm chart via OCI IstioOperator CRDs in Git Controlled chart upgrade Yes Legacy operator-only tool Operator bundle Operator-managed CRDs Operator version update Requires evaluation In practice, the less a tool depends on proprietary deployment mechanisms, the easier it is to integrate, manage, and move between Kubernetes platforms. Conclusion: the platform that gets out of the way NKP doesn't replace Kubernetes workflows—it builds on them. Helm packages applications, OCI registries distribute them, Git defines the desired state, and FluxCD keeps deployments in sync. Instead of introducing proprietary workflows, NKP brings these familiar tools together with the governance, lifecycle management, and self-service capabilities required for enterprise-scale operations. It standardizes these workflows across any environment, including public clouds, on-premises, and edge locations. For enterprise teams, the value lies in achieving consistency without sacrificing portability. As Dastmalchi-Round notes, the question isn't whether lock-in exists, but how costly it is to leave. By relying on upstream Kubernetes components, Helm charts, and GitOps workflows, organizations retain portable applications and deployment artifacts even if they choose a different platform in the future. In the end, an open platform shouldn’t be defined by its licensing model. It should be defined by how much of your platform remains yours if you decide to move on.

Running AWS resources locally is a game-changer for engineering velocity, cost optimization, and developer autonomy. Traditionally, testing cloud infrastructure required deploying directly to a staging or sandbox AWS account. This workflow introduced painful friction points: waiting for slow cloud provisioning cycles, tracking down orphaned resources that inflate the monthly bill, and requiring a constant, high-speed internet connection. LocalStack solves this by emulating core AWS services, such as S3, SQS, DynamoDB, and other services directly on your local machine inside a Docker container. When paired with Terraform, you can safely write, plan, and apply infrastructure-as-code (IaC) configuration blueprints against this local simulator. This guide walks you through the definitive "happy path" for configuring LocalStack and Terraform, followed by a robust troubleshooting handbook for common architecture-specific and container networking errors. This allows you to provision these mock resources cleanly. This allows testing Terraform code with local resources without incurring real AWS costs, requiring internet connectivity, or dealing with slow cloud provisioning cycles. The Happy Path Setup Step 1: Setting the Stage: Launching LocalStack With Docker To get started, we need our local AWS cloud stack running inside a container. We will pull the official LocalStack image, set up our credentials, and spin up the container. First, pull the latest official image to your local machine: Before firing up the container, head over to the LocalStack Web App Dashboard to grab your personal access token (PAT). While LocalStack offers an open-source community edition, advanced features or specific emulated APIs may check for a valid token. Export this token into your shell environment so the container can authenticate and activate premium features on startup: Shell export LOCALSTACK_AUTH_TOKEN="ls-..." Now, launch the container. We need to map the primary edge gateway port (4566), which routes all inbound AWS API requests, along with the standard range of ports used by individual internal services (4510-4559). We also pass our token as an environment variable: Shell docker run --rm -it \ -p 4566:4566 \ -p 4510-4559:4510-4559 \ -e LOCALSTACK_AUTH_TOKEN=$LOCALSTACK_AUTH_TOKEN \ localstack/localstack Keep an eye on your terminal logs. LocalStack will quickly validate your token, pull your license configuration, and initialize the mock runtimes. You will see a clear notification when the edge proxy is fully ready to handle incoming API requests. Step 2: The S3 Sanity Check: Talking to LocalStack Before configuring our automation toolchain, let's run a quick manual sanity check using the standard AWS CLI. Because LocalStack runs entirely on your machine, we must override the default cloud routing by passing a custom --endpoint-url pointing to our local edge proxy. To verify that LocalStack is running and reachable, create a local S3 bucket and upload a test file using the AWS CLI. 1. Create a Bucket Shell aws s3 \ mb s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 2. Upload an Object Create a dummy text file and copy it into your new mock bucket: Shell aws s3 \ cp /tmp/demo.txt s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 3. List Objects Verify the object is safely stored inside the mock container: Shell aws s3 \ ls s3://demo-bucket \ --endpoint-url=http://localhost:4566 \ --region us-east-1 Step 3: Writing the Blueprint: Configuring the Terraform Provider Now let's automate things. To instruct Terraform to deploy resources to our local simulator instead of the real AWS cloud, we must customize the AWS provider block. We enforce dummy credentials, bypass cloud-only identity validations, and explicitly force all API endpoints to route directly to http://localhost:4566. Providers Configuration Create a file named providers.tf with the following content: Markdown terraform { backend "local" { path = "terraform.tfstate" } required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = "us-east-1" access_key = "mock_access_key" secret_key = "mock_secret_key" skip_credentials_validation = true skip_metadata_api_check = true skip_requesting_account_id = true s3_use_path_style = true # Redirect all endpoints to LocalStack's edge port endpoints { apigateway = "http://localhost:4566" cloudwatch = "http://localhost:4566" dynamodb = "http://localhost:4566" ec2 = "http://localhost:4566" iam = "http://localhost:4566" lambda = "http://localhost:4566" rds = "http://localhost:4566" s3 = "http://localhost:4566" secretsmanager = "http://localhost:4566" sns = "http://localhost:4566" sqs = "http://localhost:4566" ssm = "http://localhost:4566" sts = "http://localhost:4566" } } SQS Resource Definition Next, define the SQS queue we want to provision. Create a file named main.tf: Markdown resource "aws_sqs_queue" "local_queue" { name = "my-local-queue" delay_seconds = 90 max_message_size = 2048 message_retention_seconds = 86400 receive_wait_time_seconds = 10 } output "queue_url" { value = aws_sqs_queue.local_queue.id } Step 4: The Moment of Truth: Initializing and Applying Configuration With our configuration defined, we can run Terraform. Ensure you are executing a native binary that matches your host system architecture (such as a native darwin_arm64 binary if you are working on an Apple Silicon machine) to prevent execution overhead. Initialize Terraform First, initialize the working directory to download the AWS provider plugins: Generate and Review the Plan Next, generate and review an execution plan. The plan output will detail our local queue configuration without attempting to connect to actual AWS endpoints: Apply the Plan Apply the plan to deploy the queue directly to LocalStack. Upon completion, Terraform will write your state file locally and output your new mock SQS queue URL: Step 5: Taking It for a Spin: Sending and Receiving SQS Messages To confirm that our Terraform-provisioned SQS queue is fully operational, let's capture the output URL and push a real message through it using the AWS CLI. 1. Send a Message Shell export QUEUE_URL="http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/my-local-queue" aws sqs send-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL \ --message-body "Hello from LocalStack SQS" 2. Receive the Message Shell aws sqs receive-message \ --endpoint-url=http://localhost:4566 \ --region us-east-1 \ --queue-url $QUEUE_URL { "Messages": [ { "MessageId": "1235d997-f60a-4e86-b248-aff3f5f41dde", "ReceiptHandle": "NzYxOThkMDAtMWJiOC00OGVhLTllMDEtNTU3ZTY3ZGQ5M2I4IGFybjphd3M6c3FzOnVzLWVhc3QtMTowMDAwMDAwMDAwMDA6bXktbG9jYWwtcXVldWUgMTIzNWQ5OTctZjYwYS00ZTg2LWIyNDgtYWZmM2Y1ZjQxZGRlIDE3ODI4OTI2MzcuMjg2ODc1NQ==", "MD5OfBody": "88dc2faa42b899c03e12fd3ac96d714b", "Body": "Hello from LocalStack SQS" } ] } Your terminal will return a successful JSON payload containing your message body, confirmation IDs, and MD5 hashes, proving that the local loop is entirely complete. Event Verification in LocalStack Logs Checking the LocalStack container console confirms the queue creation, message send, and message fetch operations were handled successfully: Troubleshooting Guide Even on a happy path, local container networks and mixed system architectures can throw a wrench into your workflow. Here is how to fix the most common bottlenecks. The Apple Silicon (M1/M2/M3) Rosetta Loop Symptom: The LocalStack container crashes unexpectedly on startup, or loops endlessly while attempting to launch internal components like local Lambda runtimes, throwing qemu: uncaught target signal 11 errors. The Cause: LocalStack occasionally spins up secondary processes or helper binaries inside the container. If Docker Desktop is forced to emulate an x86_64 architecture via Virtualization frameworks on an ARM64 Apple Silicon chip, the emulation layer can break during heavy nested execution. The Fix: Ensure your Docker Desktop configuration has Use Virtualization framework enabled under Settings -> General, and turn on Rosetta for x86/amd64 emulation under the Features in Development tab. Alternatively, force Docker to fetch the native ARM64 container image by updating your execution command to include the specific platform flag: Shell docker run --platform linux/arm64 --rm -it -p 4566:4566 localstack/localstack "Port Already in Use" Symptom: Docker fails to bind ports, displaying an error message like: Bind for 0.0.0.0:4566 failed: port is already allocated. The Cause: A previous instance of LocalStack didn't shut down cleanly, or another local development tool is monopolizing port 4566. The Fix: Option 1: Check for lingering Docker containers Often, a container crashed or was backgrounded but didn't release the port. Find any container using 4566: Shell docker ps -a | grep 4566 If a container shows up, stop and remove it (replace <CONTAINER_ID> with your specific ID): Shell docker stop <CONTAINER_ID> docker rm <CONTAINER_ID> Option 2: Kill native background processes If Docker isn't holding the port, another process on your host machine is. You'll need to find its Process ID (PID) and force-quit it. Find the PID: Shell lsof -i :4566 Kill it (look for the number under the PID column): Shell kill -9 <PID> Wrapping Up Combining LocalStack and Terraform gives you a lightning-fast, zero-cost, offline sandbox for cloud infrastructure development. Once your environment is configured correctly with a valid personal access token, precise Docker port mappings, and native toolchains matched to your host CPU, you can prototype, test, and tear down AWS configurations in seconds. No more waiting for slow cloud deployments or tracking down orphaned cloud resources. Happy local provisioning!

Agent framework debates are mostly vibes. One engineer swears LangGraph is faster, another prefers the OpenAI Agents SDK, someone wants Google ADK because it feels future-proof. The team picks one, wires the workflow into its SDK, and the choice is welded in. Changing frameworks later means tearing out the wiring for one SDK and rebuilding the workflow on another, an expensive rewrite few teams take on. This tutorial makes that decision reversible and then settles it with data. You put the agent graph in LaunchDarkly and run four frameworks (LangGraph, Strands, OpenAI Agents SDK, and Google ADK) over the same topology, with the model pinned so the framework is the only variable. A LaunchDarkly experiment ranks them on graph latency and token use, with an LLM judge guarding quality. The results table tells you which framework runs your graph fastest without degrading it. This tutorial is the sequel to Compare AI orchestrators, which ran the same workflow across frameworks but kept the topology in each framework’s code. Here, the topology, routing, models, prompts, tools, and judge all live in LaunchDarkly, and each framework supplies only two functions. The experiment results do more than set a benchmark. The flag that splits experiment traffic also routes production. When one framework wins, you don’t rewrite the app; you change the flag to serve the winner. In a single loop, LaunchDarkly does three jobs: the graph definition, the experiment split, and the runtime control that ships the winner. The workload is a research-gap analysis over a set of arXiv papers. Two readers, approach-analyzer and contradiction-detector, read the same papers in parallel and fan in to gap-synthesizer, which writes the report. Prerequisites A LaunchDarkly account with AgentControl access, and your environment’s SDK keyPython 3.11+ and uvAn ANTHROPIC_API_KEY for the pinned model. OPENAI_API_KEY and GOOGLE_API_KEY are only needed if you run the optional native-model bake-off in Step 9The companion repo: ai-orchestrators on branch tutorial/graph-experiments The Experiment Design The comparison is controlled: same graph, same model, same papers, same judge, with the framework as the only variable. Mechanically, it runs in four stages: Bootstrap. manifest.yaml creates the node configs, graph, orchestrator flag, and judge in LaunchDarkly.Route. On each request, the app evaluates the orchestrator flag to pick a framework: langgraph, strands, openai-agents, or google-adk.Run. The dispatcher runs the shared graph as a directed acyclic graph (DAG). The two readers run concurrently and fan in to the synthesizer.Measure. Each run records how long the graph took, how many tokens it used, and whether the report passed the quality judge. The shape looks like this: ┌──▶ approach-analyzer ───────┐ intake (papers) ─────┤ ├──▶ gap-synthesizer ──▶ report └──▶ contradiction-detector ──┘ Step 1: Create the Graph, Flag, and Judge Everything starts from one file, config/graph_experiment_manifest.yaml. It declares the fetch_paper tool, four node configs (intake plus the three agents, pinned to claude-sonnet-4-5), the graph, the orchestrator flag, and the judge. First, clone the companion repo and install its dependencies with uv: Shell git clone https://github.com/launchdarkly-labs/ai-orchestrators cd ai-orchestrators git checkout tutorial/graph-experiments uv sync Next, set up a LaunchDarkly project. The bootstrap doesn’t create one, so create it with the LaunchDarkly MCP server, the projects agent skill, or the UI. Name it graph-experiments to match the value in .env.example, so the defaults work without edits. When it exists, copy its key into LD_PROJECT_KEY and its production environment SDK key into LD_SDK_KEY in .env. The runners and experiment harness use that SDK key to evaluate the flag and graph. The bootstrap also reads LD_API_KEY from .env to create the resources. Copy the example file to create your .env: Shell cp .env.example .env # then set LD_PROJECT_KEY, LD_SDK_KEY, and LD_API_KEY in .env With the keys in place, run the bootstrap: Shell uv run python scripts/launchdarkly/bootstrap.py config/graph_experiment_manifest.yaml This creates all four node configs, the research-gap-graph, the orchestrator flag (created off), and the gap-quality-judge attached to the gap-synthesizer node (its synthesizer-claude variation, set to 100% sampling). The judge scores the final report against the source papers, so it can verify grounding and citations. A judge can only check based on the information it has, so we give it the papers, not only an upstream agent’s analysis. When the graph ships, it is incomplete by design. The bootstrap creates the contradiction-detector config but wires only intake to approach-analyzer to gap-synthesizer, leaving the detector out. You’ll add it in Step 5 to complete the parallel fan-in. When it finishes, the bootstrap prints a link to your new agent graph. Open it and review the topology before moving on. The graph shows a straight line from intake to approach-analyzer to gap-synthesizer, with contradiction-detector created but not yet wired in. Step 2: The Dispatcher Runs the Graph The dispatcher is the heart of the project, and it’s the same code for every framework. It reads the graph as a DAG, runs the entry nodes concurrently, hands every node the papers as ground truth, and connects the readers at the fan-in node. The only framework-specific pieces are build_agent and invoke, which are passed in as arguments. The whole process is about 100 lines, built on the agent graph traversal methods in the SDK. The complete dispatcher.py is in the companion repo. The dispatcher carries the design in four parts: it builds the execution plan from the graph’s edges, composes each node’s input, runs every ready node concurrently each round, and records the graph’s metrics once per run. First, the dispatcher builds the execution plan from the graph’s edges, so the topology you draw in LaunchDarkly runs: Python for key, node in nodes.items(): for edge in node.get_edges(): target = edge.target_config if target in nodes: succ[key].append(target) preds[target].append(key) Next, every node receives the source papers and any upstream analyses, so each agent and the judge work directly from the source material rather than a summary handed down a chain: Python def compose_input(user_input, predecessor_outputs): parts = [f"=== SOURCE PAPERS ===\n{user_input}"] for key, out in predecessor_outputs: if out and out.strip(): parts.append(f"=== {key} ===\n{out}") return "\n\n".join(parts) Then each round runs every node whose predecessors have finished, concurrently, so the two readers fan out and fan in with no special casing: Python ready = [k for k in pending if all(p in done for p in preds[k])] results = await asyncio.gather(*(run_node(k) for k in ready)) Finally, the dispatcher records the graph’s metrics on each run, including the end-to-end latency the experiment ranks on: Python graph_tracker.track_duration(int((time.monotonic() - start) * 1000)) graph_tracker.track_total_tokens(TokenUsage(input=totals["in"], output=totals["out"], total=totals["in"] + totals["out"])) graph_tracker.track_path(path) graph_tracker.track_invocation_success() The dispatcher reads the topology at runtime, so reshaping the workflow in the UI, adding a node, or redrawing an edge takes effect on the next request with no code change. You’ll do exactly that in Step 5. Step 3: Each Framework Is a Thin Adapter Each framework implements build_agent(node_key, config, instructions) and async invoke(agent, input_text, tracker). Everything dynamic still comes from the LaunchDarkly node config: the model, the attached tools, and the instructions. LangGraph has a LaunchDarkly companion package, so its runner is only a few lines. The companion handles model creation, tool binding, and token tracking, so the adapter holds no framework plumbing of its own: Python def build_agent(node_key, config, instructions): llm = create_langchain_model(config) tools = build_tools(config, TOOL_REGISTRY) # binds only this node's attached tools return create_react_agent(llm, tools, prompt=instructions) async def invoke(agent, input_text, tracker): result = await tracker.track_metrics_of_async( lambda res: LDAIMetrics(success=True, tokens=sum_token_usage_from_messages(res.get("messages", []))), lambda: agent.ainvoke({"messages": [{"role": "user", "content": input_text}]}), ) messages = result.get("messages", []) for message in messages: for name in get_tool_calls_from_response(message): tracker.track_tool_call(name) text = _content_to_text(messages[-1].content) if messages else "" return text, sum_token_usage_from_messages(messages) Strands has no companion package, so its runner builds the model with a small provider-aware factory and binds tools with Strands’ native @tool. The contract is identical: Python def build_agent(node_key, config, instructions): return Agent( name=node_key, model=_create_strands_model(config), system_prompt=instructions or "Process the input and respond.", tools=_bind_tools(config), callback_handler=None, ) OpenAI Agents and Google ADK round out the four. For the comparison to stay fair, all four have to run the same model, but these two SDKs default to their own vendors’ models. LiteLLM, a thin adapter, lets them call any provider, so we point both at the pinned claude-sonnet-4-5 and keep the model identical across all four orchestrators. No OpenAI or Google servers are involved. Instead, LiteLLM translates the request format in-process, and the call goes straight to Anthropic with your key. Google ADK is fully companion-free, and OpenAI Agents uses the ldai_openai companion for token and tool-call telemetry even though it builds the model through LiteLLM. This experiment pins one model across all four frameworks, so every framework here runs Claude. Pointing each framework at its own vendor’s default model instead is a separate, optional exercise, the native-model bake-off in Step 9. The tool callables live in TOOL_REGISTRY, a plain {name: callable} map that each framework binds its own way. Step 4: Smoke Test the Graph Before you run any experiment, confirm the bootstrapped graph runs end to end. First, run one framework: Python uv run python orchestrators/verify_run.py langgraph It prints the path it took and the first part of the report. On the graph as it shipped, the path is intake -> approach-analyzer -> gap-synthesizer: intake runs its short pass, approach-analyzer reads the papers, and gap-synthesizer writes the report. There’s no contradiction-detector yet, and no error. The metrics land in the AgentControl UI under the graph you created. Step 5: Add the Parallel Fan-In In the UI Here’s the payoff of keeping the topology in LaunchDarkly: you finish building the workflow in the UI, with no redeploy, and the running app picks up the new shape on its next request. The contradiction-detector config already exists, with its fetch_paper tool attached. You wire it into the graph to add the second reader and form the parallel fan-in. To complete the graph: Click Agents in the LaunchDarkly sidebar.Click Agent graphs.Select research-gap-graph.Add the contradiction-detector node.Draw an edge from intake to contradiction-detector, then another from contradiction-detector to gap-synthesizer.Click Save. You add no routing logic: the edge itself is the route, because routing is structural. Re-run the smoke test: Shell uv run python orchestrators/verify_run.py langgraph The path now includes contradiction-detector, and because approach-analyzer and contradiction-detector run concurrently, their order can vary. You completed a multi-agent workflow from the UI, and the config you wired in already had its tool attached. You finished a multi-agent workflow from the UI, mid-development, and the dispatcher ran the new shape on the next request. No redeploy, no code change: the graph you draw is the graph that runs. Step 6: Smoke Test All Four Frameworks Before you collect experiment data, make sure all four frameworks can run the completed graph. One command runs all four in sequence: Shell uv run python orchestrators/verify_run.py all It runs each framework against the completed graph and ends with a pass/fail summary, one line per framework, exiting non-zero if any framework failed, so it works as a gate. Each framework prints the path it took and a preview of its report, then a final summary collects the results. A successful run looks like this: Plain Text ▶ Running 'langgraph' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'strands' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'openai-agents' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer ▶ Running 'google-adk' over 2 papers on graph 'research-gap-graph'... ✓ PATH : intake -> contradiction-detector -> approach-analyzer -> gap-synthesizer === smoke summary === ✓ langgraph ✓ strands ✓ openai-agents ✓ google-adk If a framework fails, its line shows an ✗ instead of a ✓ and the command exits non-zero. All four smoke tests against the pinned Claude model. ANTHROPIC_API_KEY is the only model key you need, because OpenAI Agents and Google ADK reach Claude through LiteLLM. The OpenAI Agents SDK turns on tracing by default and looks for OPENAI_API_KEY to export traces, so the openai-agents run may print a harmless tracing warning when that key is absent. It doesn’t affect the run. Step 7: Run It Through the Experiment Now you can use a LaunchDarkly experiment to rank the four frameworks on real traffic, on the same graph, with the model held constant. Because the model is fixed, the comparison is operational: which orchestrator delivers the model’s quality fastest, with the least token overhead. The bootstrap already created the flag, the judge, and the graph. These metrics are measured on each request, so do a one-time setup first: Make the request context kind available for experiments.Set the analysis unit of graph latency, tokens, and the judge metric to request. Then create the experiment in the UI: Create an experiment with the orchestrator flag as the treatment.Set the primary metric to Graph latency ($ld:ai:graph:duration:total, the time for a complete graph execution).Add tokens and $ld:ai:judge:gap-quality as secondary metrics.Set the audience to 100% and the randomization unit to request. Each run is a single request, there are no users in this workflow, and request is the unit LaunchDarkly measures AI and graph metrics by.Turn on the orchestrator flag, which the bootstrap created set to off, so it serves the experiment’s variations.Start an experiment iteration. We rank on latency and tokens because, with the model and the graph held constant, those are the things that genuinely differ: a framework can move quality only by degrading the plumbing, like a truncated report or a broken tool call. So $ld:ai:judge:gap-quality stays a guardrail that catches a framework “winning” by cutting corners, not part of the ranking. Swap the model, prompt, or tools later instead of the framework, and that same judge becomes your primary metric. Then drive traffic. The flag assigns each run one framework at random: Shell uv run python scripts/run_experiment.py --runs-per-category 6 That’s six runs over each of the six shipped topics, 36 in total. Assignment is random, so it usually fills all four variations, though it isn’t guaranteed. Each run analyzes the topic’s entire paper set, because gap analysis needs every paper to find real gaps. Open the experiment in LaunchDarkly: latency per variation, with tokens and $ld:ai:judge:gap-quality alongside. The winner is the framework with the best latency and lowest token use that doesn’t let quality slip. Because the model is pinned, cost is a fixed multiple of tokens, so the token column is also the cost ranking; for actual dollar figures, read them from Insights. Because the experiment holds everything but the framework constant, most of these bars land close, often within a few percent, which is by design. In our run, Strands won on speed: it ran the graph fastest, with quality holding at the guardrail. If you optimize for speed and quality holds, that makes Strands the orchestrator to ship for this workload. Six topics and one randomized split isn’t a large sample, so confirm the lead with more topics before you standardize on it. You can do that in Step 9. Step 8: Ship the Winner With Runtime Control The experiment gave you data. The reason to run it in LaunchDarkly, rather than a one-off script, is that acting on that data takes no deploy: the orchestrator flag that was the experiment treatment is also your production router. When a variation wins, stop the iteration and set the flag’s default to that framework. Every request routes to it on the next evaluation, with no redeploy. Then automate what you don’t want to babysit. An adaptive trigger watches a guardrail and changes a flag on its own when production drifts past it. The orchestrator you shipped is operational and won’t degrade by itself, so point the trigger at the model flag from Step 9: it fails over to a backup model when your primary provider has a bad day, the same guardrail driving a different flag. That closes the loop: experiment to find the winner, runtime control to ship it, and automation to keep it healthy. Step 9: Extend the Experiment Tighten the bands by adding more topics. Confidence comes from more distinct topics, not more runs over the same few. Download one with a title-phrase (ti:) query, and the harness picks it up automatically on the next run: Shell uv run python scripts/download_papers.py --query 'ti:"LLM-as-a-judge"' Make quality the headline by flipping a config, not a flag. The framework lives in the orchestrator flag because it is app-level routing, not a property of any agent. The model, the prompt, and the tool set are different: they live in the node configs, so you experiment on the config itself. Add a second variation to a node, such as gap-synthesizer with a stronger model or a tightened prompt, and run an experiment with that config as the treatment and its variations as the arms. Pin the framework by setting the orchestrator flag to one value and leave the graph alone, so the config is the only thing moving. The judge attached to the synthesizer already emits $ld:ai:judge:gap-quality, so quality is the primary metric with no new instrumentation. Now it genuinely moves, because a different model or prompt reasons differently about the same papers. Experiment on the graph shape with a graph-key flag. The dispatcher takes the graph key as an argument, so the shape is another value you can put behind a flag: Python graph_key = ld.variation("graph_shape", context, "research-gap-graph") result = await execute_graph(ai_client, graph_key, context, user_input, build_agent, invoke) Build two graphs with different keys: for example, a linear research-gap-graph-linear (intake to approach-analyzer to gap-synthesizer) against the parallel research-gap-graph, or one with an added critic node against one without. Make a multivariate graph_shape flag whose variations are those graph keys, evaluate it exactly as the app evaluates orchestrator, and set it as the experiment treatment with the framework and model held constant. You are measuring whether the extra structure earns its latency and quality, and because the dispatcher runs whatever shape the key resolves to, no runner or dispatcher code changes. You build the judge once, and it is the guardrail for the framework bake-off, and the headline metric for every model, prompt, tool, and shape you test next. Run a native-model bake-off. This experiment holds the model constant so the framework is the only variable. To compare each framework on its own default model instead, build separate node configs per framework. This is the optional bake-off the prerequisites mention. It’s a follow-up beyond this walkthrough, and the only part that needs OPENAI_API_KEY and GOOGLE_API_KEY. Whatever you flip, follow three rules: Change one variable at a time (the framework, the model, or the shape), never two. If you change more than one, you can’t attribute the win.Keep the quality guardrail on every run, because the fastest variant is often the one that quietly truncated its report or dropped a tool call.Earn confidence with distinct inputs, not repeats: a tight band around three repeated topics is still a tight band around the wrong number. To learn more about judge design, read When to add online evals and Evaluating with LLM-as-judge evaluators. To add a pre-production regression layer, read Offline evaluation of RAG-grounded answers. Recap and Next Steps Framework choice doesn’t have to be a one-way door. Put the topology in a LaunchDarkly agent graph, have each framework supply only build_agent and invoke, and let one experiment settle a question that usually gets answered by whoever argues hardest: pin the model, let the judge guard quality, and pick the orchestrator that delivers it fastest, with evidence in hand. Then keep going, because the framework is only the first swappable component. The same flag, experiment, and judge machinery compares models, prompts, tools, and whole graph shapes the same way, so “which is better” stops being a debate and becomes a measurement. And because the experiment and the runtime control are one flag, you never stop at a finding: you ship it, ramp it with a progressive rollout, and let an adaptive trigger hold the line in production while the AI iteration loop for reliable agents keeps the next change shipping behind eval gates. The complete code is in the sample repo. Get started with AgentControl, point the four frameworks at a graph your team actually runs, and settle the next framework argument with a number instead of a hunch.

Reliable orchestration for small language models depends less on model sophistication than on the durability of event flow and state. Under the assumptions used here — small model instances, little or no local state, Kafka as the event backbone, Temporal as the orchestration layer and durable state store, and Java as the runtime — the safest design is to treat model invocations as replayable side effects, Kafka as the transport and ordering substrate, and Temporal Workflow state as the canonical record of conversational progress. In that design, Kafka provides high-throughput append-only event delivery and partition-local ordering, while Temporal persists Workflow Event History and can replay execution after failures. Exactly-once semantics remain meaningful inside Kafka’s consume-transform-produce boundary when transactions and read_committed are used, but once processing crosses into external systems such as model APIs, durable activities, or databases, correctness comes from idempotency, deduplication, sequence checks, and reconciliation rather than from a global exactly-once guarantee. Assumptions The most productive baseline is a narrow one. Each conversation, task, or model session is keyed so related events land on the same Kafka partition, preserving order only where order actually exists: within one partition, not across the topic. Each workflow instance owns one conversational state machine, stores the minimal context needed to decide the next action, and invokes model calls through Temporal Activities so failures, retries, and timeouts are visible and durable. Large prompts, attachments, or long transcripts are not kept as incidental JVM memory because Temporal persists inputs and outputs in Event History and large histories degrade replay latency; those artifacts belong in external storage with durable references held in workflow state. Analysis The central engineering mistake in LLM orchestration is to confuse transport delivery with business completion. Kafka can guarantee at-least-once delivery by processing records before committing consumer offsets, and it can provide exactly-once behavior for Kafka-to-Kafka pipelines by atomically updating produced records and consumed offsets with transactions. Kafka’s own design documentation is explicit that the producer is the transactional component and that read_committed is advisable when aiming for exactly-once processing. The same documentation also makes clear why the guarantee weakens at system boundaries: once consumed data must be coordinated with an external state store or side effect, the problem becomes cross-system consistency rather than log delivery. In a Temporal-based model pipeline, that means Kafka should usually be treated as the durable ingress path, while Temporal owns the authoritative notion of whether an event was applied to a conversation state machine. That separation suggests a simple rule. Offsets are transport progress; workflow state is semantic progress. A consumer should therefore commit offsets only after handoff to a durable semantic owner. In this architecture, that owner is the Temporal workflow receiving a signal. Temporal workflows behave like stateful services that receive Signals, Queries, and Updates, and the platform persists Event History so a crashed worker can replay the workflow and resume from the last recorded event. Signal handlers are allowed to mutate workflow state, and blocking coordination can be expressed safely with Workflow.await. Activity retries are configured through ActivityOptions and RetryOptions, with heartbeat support for long-running calls. Java @WorkflowInterface interface ModelFlow { @WorkflowMethod void run(String sessionId); @SignalMethod void onEvent(ModelEvent event); @QueryMethod long lastAppliedSequence(); } private final ModelActivities activities = Workflow.newActivityStub( ModelActivities.class, ActivityOptions.newBuilder() .setStartToCloseTimeout(Duration.ofSeconds(20)) .setRetryOptions( RetryOptions.newBuilder() .setInitialInterval(Duration.ofMillis(250)) .setMaximumAttempts(5) .build()) .build()); private final NavigableMap<Long, ModelEvent> pending = new TreeMap<>(); private long nextSequence = 1; private ConversationState state = ConversationState.empty(); @Override public void onEvent(ModelEvent event) { pending.putIfAbsent(event.sequence(), event); } @Override public void run(String sessionId) { for (;;) { Workflow.await(() -> pending.containsKey(nextSequence) || state.closed()); if (state.closed()) break; var event = pending.remove(nextSequence); state = activities.applyEvent(state, event); nextSequence = event.sequence() + 1; } Workflow.await(Workflow::isEveryHandlerFinished); } @Override public long lastAppliedSequence() { return nextSequence - 1; } This workflow fragment does three important things at once. The @SignalMethod declares asynchronous event ingress, the @QueryMethod exposes durable progress for reconciliation, and the activity stub attaches retry policy directly to the state transition that may call a model endpoint or another dependency. The pending map is not a queue for throughput; it is a reordering guard. If Kafka redelivers a message or an upstream retry arrives out of sequence, putIfAbsent and the nextSequence gate prevent semantic duplication and preserve per-session causality. Finishing the run only after Workflow.isEveryHandlerFinished() avoids the Temporal-documented failure mode where a workflow completes or continues-as-new while a handler is still waiting on asynchronous work. The matching Kafka consumer must be deliberately conservative. Automatic commits are inappropriate because they advance transport progress in the background regardless of semantic application. Manual synchronous commits make the boundary explicit, and Kafka documents that committed offsets are the secure restart position, and that commitSync should write the next offset, meaning lastProcessedOffset + 1. The consumer is also not thread-safe, so per-partition in-order handling is easiest when one poll loop owns one consumer instance and performs durable handoff before commit. Java void pollLoop() { consumer.subscribe(List.of("model-events")); while (running.get()) { var records = consumer.poll(Duration.ofSeconds(1)); for (var partition : records.partitions()) { var batch = records.records(partition); for (var record : batch) { var eventId = header(record, "event-id"); if (!inbox.tryInsert(eventId, record.topic(), record.partition(), record.offset())) { continue; } var workflow = client.newWorkflowStub(ModelFlow.class, record.key()); workflow.onEvent(ModelEvent.from(record)); inbox.markApplied(eventId); } var nextOffset = batch.get(batch.size() - 1).offset() + 1; consumer.commitSync(Map.of(partition, new OffsetAndMetadata(nextOffset))); } if (inbox.backlog() > 50_000) consumer.pause(consumer.assignment()); else consumer.resume(consumer.assignment()); } } The durable inbox is the effective-once bridge. If the process crashes after signaling Temporal but before committing offsets, Kafka may redeliver, yet tryInsert suppresses reapplication. If upstream producers use Kafka transactions, the consumer should read with isolation.level=read_committed so aborted records stay invisible; Kafka’s configuration reference notes that read_committed returns only committed transactional messages and withholds records past the last stable offset while open transactions exist. Backpressure also belongs here. Kafka exposes pause and resume without forcing a group rebalance, and monitoring guidance explicitly recommends watching lag, fetch rate, poll timing, and commit latency to ensure consumers are keeping up. Context propagation is easiest when context is split into stable metadata and mutable conversational state. Stable identifiers such as trace ID, tenant, policy version, and conversation key belong in Kafka headers and Temporal headers so they survive hops across services and activities; Kafka’s ProducerRecord supports headers, and Temporal context propagators move custom key-value data across workflow, activity, and child-workflow boundaries. Mutable context, by contrast, should not live in worker memory or ad hoc caches. It belongs in the workflow state, often as a compact summary plus references to offloaded artifacts. Temporal’s documentation explicitly warns that all activity inputs and outputs are persisted, that long AI-style conversations grow history, and that large histories degrade workflow-task latency. For long-running sessions, Continue-As-New provides a checkpoint boundary with a fresh Event History while preserving the workflow identity chain. Reconciliation closes the last reliability gap. Even with careful commits, outages, manual replays, or producer bugs can create suspicion that a workflow missed an event. Temporal queries are read-only and must not mutate state or block, which makes them ideal for asking a workflow for its durable high-water mark and replaying any gap from the event store. Java void reconcile(String workflowId, long durableHighWatermark) { var workflow = client.newWorkflowStub(ModelFlow.class, workflowId); long applied = workflow.lastAppliedSequence(); eventStore.readRange(workflowId, applied + 1, durableHighWatermark) .forEach(workflow::onEvent); } This pattern works because the workflow does not trust delivery history alone; it trusts its own durable state. Observability then becomes the enforcement layer for those guarantees. Kafka should surface lag, request latency, retry rates, poll gaps, and buffer exhaustion, while Temporal should emit metrics through Micrometer, trace activity and workflow paths, and expose searchable workflow metadata through Search Attributes. Temporal also recommends monitoring replay latency because large histories, payload sizes, and cache churn drive recovery cost. Together, these signals reveal the difference between a system that is slow, a system that is duplicating work, and a system that is actually losing context. Conclusion Orchestrating small language models without losing events or context is fundamentally a durability problem, not a prompt-engineering problem. Kafka should be used for ordered transport and scalable ingestion, but semantic completion should be anchored in Temporal’s durable workflow state, where signals, sequence gates, retryable activities, queries, and replay make failures recoverable rather than ambiguous. Exactly-once remains valuable inside Kafka’s transactional envelope, yet end-to-end correctness across model calls and other side effects comes from explicit idempotency, deduplication, reconciliation, and bounded context management with external storage and continue-as-new. In a Java stack, that combination yields an architecture where duplicates become harmless, ordering becomes explicit, back pressure becomes controlled, and context survives crashes because it is recorded in durable state instead of being left in process memory.

By Akhil Madineni CORE

This guide explains zone-aware routing from a Kubernetes-first point of view. It covers: why zones matter in cloud platformswhich topology labels Kubernetes places on nodeshow Kubernetes first tried to solve locality through Servicewhat gaps remained after those Service-based featureshow Gateway API implementations such as Envoy Gateway and kgateway built on top of that foundation Why Zones Matter In cloud platforms, a zone is a logical failure domain inside a region. Zones usually have low-latency networking within the zone, but crossing zones can increase both latency and cost. That cost is not theoretical. AWS documents that traffic within the same Availability Zone is free, while traffic that crosses Availability Zones typically incurs data transfer charges, and cross-zone transfer is generally billed in both directions, so a single round trip can be charged twice. See: AWS Architecture Blog: Overview of Data Transfer Costs for Common ArchitecturesAmazon EC2 pricing: Data Transfer This is one reason distributed systems try to keep traffic local when they can, while still preserving failover to other zones. The Topology Information Kubernetes Already Has Kubernetes did not start by inventing zone-aware traffic policies. It started by carrying topology information on nodes. The two most important well-known labels are: topology.kubernetes.io/regiontopology.kubernetes.io/zone According to the Kubernetes reference, these labels are populated on Node objects by the kubelet or the external cloud-controller-manager when the cluster is integrated with a cloud provider. In non-cloud environments, operators can set them manually if the topology model still makes sense. Reference: Kubernetes well-known labels: topology.kubernetes.io/zone In managed clusters, these labels are commonly present by default. Here is the kind of node data Kubernetes typically exposes: YAML apiVersion: v1 kind: Node metadata: name: ip-10-0-12-34.ec2.internal labels: kubernetes.io/hostname: ip-10-0-12-34.ec2.internal topology.kubernetes.io/region: us-east-1 topology.kubernetes.io/zone: us-east-1a That topology data is useful for scheduling, spreading replicas, volume placement, and eventually traffic routing. The Original Service Model The original Kubernetes Service abstraction solved a different problem first: stable discovery and virtual IPs for ephemeral Pods. At the beginning, the model was simple: a Service selected a set of Podskube-proxy programmed forwarding rulestraffic could be sent to any healthy endpoint behind the Service That was excellent for reachability and abstraction, but it had no built-in notion of zone locality. The gap was straightforward: the Service abstraction knew which endpoints existed, but not that a client in zone-a should usually prefer endpoints in zone-a. Kubernetes' First Attempts to Improve Locality Through Services Kubernetes gradually added locality-aware behavior on top of Service, mostly by improving how endpoint selection works. Internal Traffic Policy One early mechanism was internalTrafficPolicy: Local. This tells kube-proxy to use only node-local endpoints for cluster-internal traffic. Example: YAML apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 internalTrafficPolicy: Local Reference: Kubernetes Service Internal Traffic Policy This helps with node locality, but it is not zone-aware routing. Its limitations are important: it is node-local, not zone-localif a node has no local endpoint, the Service behaves as if it has zero endpoints from that node's perspectiveit is too strict for many multi-zone workloads that want zonal preference, not node affinity So this was useful, but it did not really solve multi-zone locality. Topology Aware Routing With Services Kubernetes next introduced Topology Aware Hints, now called Topology Aware Routing. This works through two components: The EndpointSlice controller looks at endpoint and node topology.kube-proxy consumes hints from EndpointSlices and prefers endpoints closer to the client zone. Historically, the Service-side configuration was commonly exposed through the service.kubernetes.io/topology-mode: Auto annotation: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend annotations: service.kubernetes.io/topology-mode: Auto spec: selector: app: backend ports: - port: 80 targetPort: 8080 Conceptually, the flow looks like this: This was Kubernetes' first real zone-aware answer at the Service layer. It is useful historical context, but it is no longer the clearest Service-level API to emphasize for new users. Traffic Distribution Preferences Kubernetes later added trafficDistribution as a clearer way to express routing preferences. In current Kubernetes documentation, the relevant zone-level preference is: PreferSameZone The older PreferClose name is documented as deprecated in favor of PreferSameZone, though you may still see PreferClose in some provider and implementation docs that have not yet caught up. Example: YAML apiVersion: v1 kind: Service metadata: name: zone-aware-backend spec: selector: app: backend ports: - port: 80 targetPort: 8080 trafficDistribution: PreferSameZone Reference: Kubernetes Service trafficDistribution This is a better API shape than older annotations because it is explicit in the Service spec and described as a preference rather than a strict guarantee. In practice, that means current Kubernetes guidance emphasizes trafficDistribution: PreferSameZone, while the older topology-mode: Auto path is best understood as part of the feature's evolution. What Gap Remained After Service-Based Locality Kubernetes Services improved a lot, but they still left several gaps. The Behavior Is Best Effort Topology-aware routing is not a hard guarantee. Kubernetes documents multiple safeguard cases where the system falls back to cluster-wide routing. Examples include: too few endpointsimpossible balanced allocationmissing topology labels on one or more nodesmissing hints for one or more endpointsno hinted endpoint for the local zone That is correct for safety, but it means the behavior is heuristic and conditional. It Assumes a Certain Traffic Shape Kubernetes explicitly documents that Topology Aware Routing works best when traffic is roughly evenly distributed and when there are enough endpoints per zone. If most traffic originates from one zone, local subsets can overload while the global service still looks healthy. It Is Scoped to the Service Datapath This is the most important architectural gap. Service-level topology features influence how kube-proxy chooses endpoints for Service traffic. They do not automatically solve every higher-level data plane. In particular, they do not by themselves define: how an L7 gateway proxy should understand its own zonehow an Envoy-based gateway should configure locality-aware upstream load balancinghow a gateway controller should express stricter local preference versus simple best-effort localityhow policy should attach to particular routes, gateways, or backends That left room for Gateway API implementations to expose richer locality controls. Why Gateway API Implementations Stepped In Gateway API is intentionally expressive and extensible. It standardizes core routing objects, but implementations often add policy CRDs to expose features that are specific to their data plane. That distinction matters here: Gateway API itself does not define one universal, cross-implementation zone-aware policy. Instead, it gives implementations room to expose locality behavior in a way that matches their proxy and control-plane design. Reference: Gateway API overview This is where zone-aware routing became more explicit at the gateway layer. Instead of relying only on kube-proxy's Service behavior, gateway implementations can: understand the proxy's own localityread backend endpoint localityconfigure the underlying proxy's load balancer directlyexpose locality policies as route or backend-attached configuration Example of How Envoy Gateway Addresses the Gap Envoy Gateway supports two paths: Reusing Kubernetes Service-level locality such as Topology Aware Routing or trafficDistributionConfiguring zone awareness directly through BackendTrafficPolicy Reference: Envoy Gateway zone-aware routingEnvoy zone-aware routing Example BackendTrafficPolicy: YAML apiVersion: gateway.envoyproxy.io/v1alpha1 kind: BackendTrafficPolicy metadata: name: zone-aware-routing spec: targetRefs: - group: gateway.networking.k8s.io kind: HTTPRoute name: zone-aware-routing loadBalancer: type: RoundRobin zoneAware: preferLocal: minEndpointsThreshold: 1 force: minEndpointsInZoneThreshold: 1 That is a meaningful step beyond plain Service because the gateway layer is now explicitly participating in locality-aware upstream balancing. Example of How kgateway Addresses the Gap kgateway takes a similar approach in spirit: proxy locality is made explicit, and backend load-balancing behavior is configured through policy rather than relying only on Service heuristics. At a high level, kgateway combines: Gateway proxy locality configurationBackend-attached load-balancing policyNative Envoy locality-aware upstream load balancingEndpoint locality metadata that Envoy can use directly Architectural Summary The progression looks like this: Kubernetes Service solved stable discovery and reachability.internalTrafficPolicy improved node-local routing, but not zonal routing.Topology Aware Routing and trafficDistribution added zone-aware preferences to the Service datapath.Gateway API implementations extended the model so L7 gateways and proxies could make explicit locality-aware decisions themselves. Practical Takeaways Kubernetes already provides the topology metadata needed for zone-aware decisions.Service-native locality is useful, but it is heuristic and scoped to the Service datapath.Zone-aware traffic for gateways usually needs the gateway implementation to understand locality too.Modern Gateway API implementations fill that gap by attaching locality-aware load-balancing policy closer to the L7 data plane. Where Zone-Aware Routing Matters in Practice Zone-aware routing usually becomes worth the added operational attention when one or both of these are true: The workload has a tight latency budget, especially at p95 or p99The system moves enough east-west traffic that even a small per-GB cross-zone charge becomes material Common examples include: Gaming platforms, where matchmaking, player session state, inventory, and real-time coordination are sensitive to a few extra milliseconds of network delayFinancial services, where payment, quote, fraud, or checkout paths care more about predictable tail latency than average latencyLarge SaaS and enterprise control planes, where a gateway fans out to many internal APIs and the aggregate cross-zone traffic becomes a real monthly costAI inference, media delivery, logging, and telemetry pipelines, where payload sizes are large enough that bandwidth cost matters even when latency is less critical Worked Example: Multiplayer Gaming Backend Suppose a regional game API runs gateway proxies and backend pods in three zones. Players connect to a gateway in zone-a, and that gateway calls a player-state service that is also deployed in zone-a, zone-b, and zone-c. Assume the following: 25,000 requests per second reach the player-state service from zone-athe combined request and response payload is about 40 KiB per callcross-zone traffic is billed at a representative $0.01 per GBwithout zone awareness, only about one third of those calls stay in zone-a, while the other two thirds go to zone-b or zone-c Actual billing varies by provider, region, and direction of transfer, but the point of the example is that a seemingly small per-GB rate compounds quickly on hot service paths. That means the traffic volume from zone-a to the player-state service is about: 25,000 x 40 KiB per second, or roughly 1 GB/s totalif two thirds of that traffic crosses zones, that is about 0.67 GB/s of cross-zone trafficover a 30-day month, that is about 1.7 million GBat $0.01 per GB, that is about $17,000 per month in cross-zone transfer for just that one service path That is the cost side. The latency side can matter even more for the player experience. If each cross-zone hop adds only 1-3 ms, a request path that fans out to several internal services can add multiple milliseconds of extra tail latency. For a gaming workload, that can affect: matchmaking responsivenesssession join timethe smoothness of player state or presence updateshow stable the system feels during traffic spikes and retries This is why zone-aware routing is not only a cost optimization. In some industries, it is a user-experience and SLO control. Worked Example: Large SaaS Control Plane The same logic applies outside gaming. Consider a large enterprise SaaS platform where each incoming API request hits a gateway and then fans out to an auth service, tenant metadata service, feature-flag service, and audit pipeline. Even if each individual backend call is small, the gateway can generate a large amount of aggregate east-west traffic. In that kind of system, zone-aware routing helps in two ways: it removes avoidable cross-zone traffic from the steady-state hot pathit reduces the chance that a multi-hop request burns several extra milliseconds just on internal network distance For that kind of platform, the business case is usually a combination of lower regional data-transfer cost, tighter latency distributions, and better failure-domain alignment. Conclusion Zone-aware routing is the story of a single idea moving down the stack. Kubernetes started with topology labels on nodes, then taught the Service datapath to prefer local endpoints through internalTrafficPolicy, Topology Aware Routing, and trafficDistribution. Those features are valuable, but they are best-effort and they stop at the Service boundary, which leaves L7 gateways unable to reason about their own locality. Gateway API implementations such as Envoy Gateway and kgateway pick the idea up from there, making proxy locality explicit and pushing locality-aware load balancing into Envoy where it can act on real endpoint metadata. The practical guidance is short. Start with the Service-native controls, because they are simple and often enough. Reach for gateway-level locality policy when you have a tight tail-latency budget, or enough east-west traffic that cross-zone transfer becomes a line item you can see. In both cases, the goal is the same: keep traffic local when you safely can, and fail across zones when you must. Further Reading Kubernetes ServiceKubernetes Topology Aware RoutingKubernetes Service Internal Traffic PolicyKubernetes well-known topology labelsGateway API overviewAWS Architecture Blog: Data transfer costs

AWS can send audit logs to an attacker’s account unless denials are enforced at the network layer, while Azure doesn’t log network-block requests at all. The concept of a data perimeter was popularized by AWS [1] to establish organizational boundaries around identities, resources, and networks. In simple terms, AWS provides access controls to ensure that trusted identities access trusted resources from expected networks while blocking all outside access. This article explores how different cloud providers handle resource access logs and how it relates to data protection. It sets up an experiment where an outside identity with valid credentials accesses a trusted resource and is blocked by a policy in one of the scenarios. The experiment explains two scenarios that differ in where the deny decision is enforced. We find that the same request for resource access produces different log artifacts in AWS and Azure. AWS sends access logs containing caller-controlled metadata in both the identity and resource-owner accounts unless a network layer explicitly denies access. However, in Azure, resource access logs are only logged at the resource-owner’s subscription, and when access is blocked at the network layer, nothing is logged there either. Both behaviors have consequences for security teams collecting and analyzing audit logs. This article walks through both scenarios with lab experiments and reproducible code. Background AWS and Azure treat identities differently. In AWS, identities are not centralized into one single place — instead, they live at the account level. For example, if an organization contains 10 accounts, identities can be created in each of the 10 accounts. In comparison, in Azure, identities are centralized into one Entra ID tenant. Since a tenant is linked to multiple subscriptions containing the company’s resources, identities from the same tenant are configured to access resources inside subscriptions. In summary, the resource-owning entity in AWS (the account) also holds identities, whereas in Azure the resource-owning entity (the subscription) does not hold identities – those live in the Entra ID tenant. Secondly, AWS and Azure treat access logging differently. In AWS, CloudTrail logs API calls at the account level. For cross-account access, AWS lets customers configure CloudTrail such that when data events are enabled, the caller account and the resource-owning account get access events. For example, if an identity in Account-A accesses a resource in Account-B and gets denied, then the deny audit entry is logged in both Account-A and Account-B. This mirroring is what makes caller-controlled metadata visible to a malicious actor’s account [2]. In contrast, in Azure, resource access logs (for example, StorageBlobLogs) live in the storage account in the subscription, whereas identity logs (Entra ID) live with the tenant. These are separate systems with no automatic mirroring. This difference sets up why a correlation problem exists and why a network-layer block does not produce logs at the resource layer. Threat Model The threat model is as follows: an attacker brings their credentials inside a corporate network and accesses the company’s resource (like an S3 bucket). By doing this, the attacker tries to exfiltrate company data by encoding sensitive information in the HTTP user agent header, a caller-controlled field that appears in access logs. This allows data to leave the corporate environment in small chunks across multiple requests. The second threat is more nuanced. A security team that relies on resource-layer logs to detect unauthorized access attempts will miss requests that are blocked before reaching the resource. If the network drops the request silently, the resource (service) never logs it. An attacker who knows this can probe a corporate environment repeatedly without appearing in the audit trail that the security team is monitoring. Experiments AWS Experiment To set up this experiment, we have three accounts: a credential-owning account (identity), a VPC-owning account, and a resource-owning account. The identity is a Lambda function that tries to access an S3 bucket (resource). The Lambda function runs from a private subnet in a VPC and accesses the S3 bucket through an S3 VPC endpoint (AWS PrivateLink). All audit logs are sent to a third account – this is a typical Control Tower setup [3]. We test two scenarios: The bucket policy denies all untrusted identities — assume that the bucket policy denies access to our identity. However, the VPC endpoint policy allows all cross-account access. The bucket policy allows this untrusted identity. However, the VPC endpoint policy disallows cross-organization access. Scenario 1 When the request gets denied at S3, AWS CloudTrail generates a standard API event: JSON { "eventType": "AwsApiCall", "errorCode": "AccessDenied", "userAgent": "...", "requestParameters": {...}, "tlsDetails": {...} } The full log is in https://github.com/sureshgururajan/aws-data-exfiltration-demo/blob/main/testing-results/scenario1-log.md. In this case, the full request context is preserved. This includes: userAgent requestParameters TLS metadata Additional request context The main observation is that this event includes caller-controlled metadata in the userAgent field. Since customers can configure CloudTrail to log data events on both the caller account and the resource account, a malicious actor gets the same denial event in their account. Therefore, an attacker in an untrusted account can exfiltrate company data into their accounts by triggering these denied access requests on the company resource. Scenario 2 In the second scenario, if the VPC endpoint policy denies cross-account access (example), CloudTrail generates a different event: JSON { "eventType": "AwsVpceEvent", "eventCategory": "NetworkActivity", "errorCode": "VpceAccessDenied", ... } See the full log here. Instead of logging an AwsApiCall event, CloudTrail logs NetworkActivity with the errorCode: VpceAccessDenied and does not log the HTTP user agent header. More importantly, this event is not sent to the malicious actor or the resource owner’s account. Rather, the event is sent to the VPC endpoint owner’s account. In other words, the cause of the denial was a VPC endpoint policy, and therefore CloudTrail generates a NetworkActivity event rather than the API event and routes it to the VPC-owning account. This prevents the bad actor from stealing company data via CloudTrail. Azure Experiment To set up this experiment, we created two Azure subscriptions – one for identity and the other for the resource. An Azure function in subscription-A writes to a blob storage in subscription-B. The Azure function is registered as a system-assigned managed identity in the Entra ID tenant while turning off the shared access key for the blob storage to ensure only managed identities can access it [5]. The function uses DefaultAzureCredential to request a token from Entra ID and attempts to write to a file in the storage account. Since both subscriptions trust the same Entra ID tenant, the identity moves across subscriptions natively without needing an AssumeRole step. Like before, we run through two scenarios: Azure function has the Storage Blob Data Contributor role and the network path is open The Azure function attempts to write to the storage account but is blocked by the firewall. Scenario 1 When the request is allowed at the blob storage, the following logs are written: The Entra ID tenant gets a token request log when the Azure function uses default Azure credentials. This event does NOT contain any information about the actual API action being taken. The resource account StorageBlobLogs records a PutBlob event with the file name and IP address but doesn’t show the name of the managed identity. Sample log entry from StorageBlobLogs Plain Text TimeGenerated [UTC] - 2026-05-02T19:30:32.7306109Z OperationName - PutBlob CallerIpAddress - 172.24.1.71:9156 Uri - https://sgrstorageaccountinsubb.blob.core.windows.net:443/storage-container/test.json AuthenticationType - OAuth RequesterObjectId - 00daa177-96c6-4b29-9a5c-53ca603565e9 StatusCode – 201 UserAgentHeader - azsdk-js-azure-storage-blob/12.31.0 core-rest-pipeline/1.22.3 Node/22.22.2 (Linux 6.6.130.1-3.azl3; x64) The requester object ID field indicates which identity made the request but doesn’t reveal more details as to the identity itself. That part is left to the Entra ID logs as shown below. However, we can see that the userAgentHeader is logged. The difference with AWS is that in Azure, the StorageBlob log entry is not mirrored to Entra ID, i.e., the caller’s subscription. In Azure, it stays only in the resource owner’s subscription. Entra ID contains just the token issuance log: Sample log entry from Entra ID Plain Text Date (UTC),2026-05-02T19:30:32Z Request ID,25c5f7f7-4206-448d-817b-730744991701 Correlation ID,73cf7b90-c49b-40f0-800d-74e77e40717c Service principal ID,00daa177-96c6-4b29-9a5c-53ca603565e9 Service principal name,SureshTestingMultiCloud-Function Credential key ID, Credential thumbprint, Application,SureshTestingMultiCloud-Function Application ID ,57650788-dae5-416f-9da8-792b4ebbbb29 App owner tenant ID, Resource,Azure Storage Resource ID ,e406a681-f3d4-42a8-90b6-c2b029497af1 Resource tenant ID, Resource owner tenant ID,f8cdef31-a31e-4b4a-93e4-5f571e91255a Home tenant ID, Home tenant name, IP address, Location,", , " Status,Success Sign-in error code, Failure reason,Other. Conditional Access,Not Applied Scenario 2 In this scenario, we introduced a network-level block using the Storage Account Firewall while keeping the permissions intact. Entra ID logs still show a successful token issuance because the identity is valid and the scope is broad. However, the storage resource logs don’t log the request. Since the connection was dropped at the network layer before reaching the storage service plane, there is no “Access denied” event in the resource’s audit log. Sample log entry from Entra ID Plain Text Date (UTC): 2026-05-02T19:35:10Z Service principal name: SureshTestingMultiCloud-Function Application: SureshTestingMultiCloud-Function Resource: Azure Storage Status: Success Sample log entry from StorageBlobLogs 0 results for the KQL query: SQL // Query to check for any recorded activity after the network block StorageBlobLogs | where TimeGenerated > ago(1h) | where RequesterObjectId == "00daa177-96c6-4b29-9a5c-53ca603565e9" | project TimeGenerated, OperationName, StatusCode, StatusText, CallerIpAddress, Uri | sort by TimeGenerated desc This result shows that a network-level block is not visible in the resource layer. The Azure administrator sees a successful token issuance in Entra ID but nothing in StorageBlobLogs. To detect this, security teams need to go beyond resource-layer logs and enable additional logging layers such as NSG Flow logs or Defender for Storage - these are outside the scope of this experiment. Comparison scenarioawsazure Identity model Account-scoped Tenant scoped Who gets audit logs? (when available and enabled) Caller-side and resource-owner side (Scenario 1 only) Resource-owner side only Where are the audit trails located? CloudTrail is the logging service. CloudTrail logs are distributed across Caller account, the resource account, and the VPC-owning account Token issuance logs are in the Tenant (Entra ID) while resource access logs are in the Subscription Caller-controlled metadata visible? Yes, visible in caller account and resource account Yes, but included in resource account only What a network-layer block produces When using VPC endpoint policy, AwsVpceEvent is produced and is routed to the VPC-owner account. No logs in resource-owner account. No resource-layer log entry. Identity context in resource logs Full caller identity context included Only the caller ID in the form of RequesterObjectId. An operator must correlate this ID with service principal ID in Entra ID logs. Mitigation We saw that in AWS, CloudTrail can be configured to send log events on both the caller account and the resource account. An attacker can use this information to silently exfiltrate small amounts of data at a time. To mitigate this attack vector, an organization must: Run their compute services in an Amazon VPC — preferably in a private subnet, and Use VPC endpoints with endpoint policies [4] to access their AWS resources for the compute services. The endpoint policies must allow trusted identities to access the resource while blocking everything else. AWS already documents these controls in [1], but these experiments show how important it is to enforce these controls. This is in addition to all the controls that an organization already uses, such as Service Control Policies and Resource Control Policies — those policies control the maximum permissible action that can be taken by an identity/resource but do not control the CloudTrail logging behavior. While Azure doesn’t have the above attack vector specifically, it has a different problem — an operator must manually correlate Entra ID events with the resource event. An example would be an “identity journey” like — managed identity (like the Azure function) requests a token, then writes to a storage account. Therefore, some tooling must be built to correlate such events — for example, routing both ManagedIdentitySignInLogs and StorageBlobLogs into a single Log Analytics workspace is a minimum. Additionally, logs must be captured at different layers such as NSG flow logs/Defender for Storage that can provide anomaly detection beyond standard diagnostic logs. Conclusion In this article, we demonstrated how the same access request produces different results in AWS and Azure. In AWS, access logs were sent to the resource account or the VPC account depending on where the deny decision was enforced, while in Azure, access logs were only sent to the resource account. We saw that this difference comes from how each cloud provider fundamentally treats identities and resources. The implications of the experiment are that security teams in multi-cloud environments cannot assume that audit coverage works the same way across providers. Each provider models their identities and provides different data perimeter controls. Before designing data perimeter controls, security teams must understand each provider’s logging architecture and its differences. References [1] https://aws.amazon.com/identity/data-perimeters-blog-post-series/ [2] https://systemweakness.com/a-subtle-audit-log-consideration-in-aws-063752150b20 [3] https://docs.aws.amazon.com/controltower/latest/userguide/what-shared.html [4] https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html [5] https://learn.microsoft.com/en-us/azure/storage/common/shared-key-authorization-prevent?tabs=portal

The Model Context Protocol connects AI agents to your databases, APIs, and file systems. Out of the box, it connects them with no identity, no scoping, and no audit trail. The MCP specification acknowledges this gap explicitly. Its OAuth 2.1 authorization spec marks authentication as optional. The result, according to research published on Security Boulevard in April 2026, is that 53 percent of open-source MCP implementations ship with static API keys. Eighty-eight percent require backend authentication, but only 8.5 percent implement proper credential management. Every one of those static keys is a credential waiting to be stolen, a scope waiting to be abused, and an audit entry that will read "unknown agent executed query" when the incident report is written. This article builds the alternative. We will build an MCP server in Python that accepts tool calls only from authenticated agents, validates OAuth 2.1 Bearer tokens using stateless JWKS-based validation, enforces tool-level scopes and roles, maintains an infrastructure-level tool allow-list, and logs every access decision with the full delegation chain back to the human who authorized it. The complete companion project, roughly 350 lines of Python with a 13-test suite, is available on GitHub. Prerequisites You will need Python 3.12 or later and an OIDC-compatible identity provider. The examples use Auth0 (free tier works), but Okta, Keycloak, Entra ID, or any provider that exposes a /.well-known/jwks.json endpoint will work. Basic familiarity with OAuth 2.1 concepts and MCP server architecture is assumed. All code shown is extracted from the companion project. File paths reference code/src/. Architecture Every tool call flows through five gates before reaching your business logic: Architecture: Five-gate MCP tool call authorization pipeline. Gates two and three are infrastructure-level controls. System prompts are not security controls. An MCP server the agent has not been explicitly authorized to call should be unreachable. Period. Regardless of what the LLM decides to invoke. Part 1: JWKS-Based Token Validation The foundation of an identity-aware MCP server is stateless JWT validation. Every request carries a Bearer token issued by your OAuth 2.1 authorization server. The MCP server validates it against the provider's JSON Web Key Set, a public key document that lets you verify signatures without a network call to the IdP on every request. The JWKS Cache Create src/auth/middleware.py. We start with a cache that fetches the JWKS once and holds it in memory, refreshing every five minutes or on-demand when an unknown key ID appears (key rotation): Python class JWKSCache: """Cached JWKS with automatic refresh on unknown key id.""" def __init__(self, jwks_url: str, cache_ttl: int = 300): self._url = jwks_url self._ttl = cache_ttl self._keys: dict[str, dict] = {} self._last_fetch: float = 0 async def get_key(self, kid: str) -> dict: if not self._keys or (time.monotonic() - self._last_fetch) > self._ttl: await self._refresh() key = self._keys.get(kid) if key is None: logger.info("Unknown kid '%s', forcing JWKS refresh", kid) await self._refresh() key = self._keys.get(kid) if key is None: raise AuthError(f"Key '{kid}' not found in JWKS", 401) return key async def _refresh(self) -> None: if self._url.startswith("http"): async with httpx.AsyncClient() as client: resp = await client.get(self._url, timeout=10) resp.raise_for_status() jwks = resp.json() else: with open(self._url) as fh: jwks = json.load(fh) self._keys = {k["kid"]: k for k in jwks.get("keys", [])} self._last_fetch = time.monotonic() The get_key method is where the key rotation logic lives. When a token arrives with a kid the cache has never seen, we force a refresh before rejecting it. An unknown kid could mean a legitimate rotation, not an attack. We try once more before failing. In practice, this means you never need to restart your MCP server when your identity provider rotates signing keys. The Token Validator The validator uses the cache to verify every Bearer token. It checks five things, and the order matters: header validity, signature, issuer, audience, and expiry: Python class TokenValidator: def __init__(self, jwks_url: str, issuer: str, audience: str, clock_tolerance: int = 30): self._jwks = JWKSCache(jwks_url) self._issuer = issuer self._audience = audience self._clock_tolerance = clock_tolerance async def validate(self, token: str) -> ValidatedToken: # 1. Decode header to get the key id. unverified = jwt.get_unverified_header(token) kid = unverified.get("kid") if not kid: raise AuthError("Token header missing 'kid' claim", 401) # 2. Fetch the matching public key. jwk = await self._jwks.get_key(kid) # 3. Verify signature + standard claims. claims = jwt.decode( token, jwk, algorithms=["RS256"], issuer=self._issuer, audience=self._audience, options={"verify_exp": True, "require": ["exp", "iss", "sub", "aud"]}, ) # 4. Clock-tolerance check (belt-and-suspenders with the library). now = int(time.time()) if claims["exp"] + self._clock_tolerance < now: raise AuthError("Token has expired", 401) # 5. Extract scopes, roles, and delegation chain. scope_str = claims.get("scope", "") token_scopes = set(scope_str.split()) roles = claims.get("roles", []) delegation_chain = self._extract_delegation(claims) return ValidatedToken( subject=claims["sub"], email=claims.get("email"), roles=roles, scopes=token_scopes, delegation_chain=delegation_chain, ) The iss (issuer) check prevents tokens from a different authorization server from being accepted. The aud (audience) check prevents tokens intended for a different service from being replayed against yours. The exp check with clock tolerance handles the reality that clocks drift. Thirty seconds of tolerance is the pragmatic default recommended by the Upstash MCP OAuth deep-dive. The delegation chain extraction is worth examining separately. When an agent acts on behalf of a human who authorized it, RFC 8693's act claim carries that nesting. We recursively unpack it: Python def _extract_delegation(self, claims: dict) -> list[str]: chain = [] act = claims.get("act", {}) while act: sub = act.get("sub", "") if sub: chain.append(sub) act = act.get("act", {}) return chain A token issued directly to a human will have an empty delegation chain. A token issued to an agent acting on behalf of "[email protected]" will carry ["[email protected]"]. A multi-hop chain, human to orchestrator agent to sub-agent, carries both identifiers in order. This is what lets your audit logs trace every action back to a person. Part 2: The Two Mandatory Discovery Endpoints An MCP client connecting to your server needs to discover two things: that authentication is required, and where to get tokens. The MCP specification mandates two well-known endpoints for this, defined in RFC 9728 and RFC 8414, respectively. Create src/auth/discovery.py: Python def build_discovery_routes( resource_url: str, authorization_server_url: str, scopes_supported: list[str] | None = None, ) -> dict: async def protected_resource(request: Request) -> JSONResponse: return JSONResponse({ "resource": resource_url, "authorization_servers": [authorization_server_url], "bearer_methods_supported": ["authorization_code"], }) async def authorization_server(request: Request) -> JSONResponse: return JSONResponse({ "issuer": authorization_server_url, "authorization_endpoint": f"{authorization_server_url}/authorize", "token_endpoint": f"{authorization_server_url}/oauth/token", "jwks_uri": f"{authorization_server_url}/.well-known/jwks.json", "scopes_supported": scopes_supported or [ "database.read", "database.write", "email.send", "admin.users.read", ], "response_types_supported": ["code"], "grant_types_supported": ["authorization_code", "client_credentials"], "code_challenge_methods_supported": ["S256"], "token_endpoint_auth_methods_supported": ["none"], }) return { "/.well-known/oauth-protected-resource": protected_resource, "/.well-known/oauth-authorization-server": authorization_server, } Without these endpoints, MCP clients cannot auto-discover your authentication configuration. The client first hits your server without a token, receives a 401 with a WWW-Authenticate header pointing to the protected resource metadata, fetches it to confirm auth is required, then reads the authorization server metadata to learn the token endpoint and supported grant types. code_challenge_methods_supported: ["S256"] is not optional. MCP clients are public clients. They cannot keep a client secret, so PKCE is the only defense against authorization code interception. The NAPTHA AI reference implementation explicitly documents this. Part 3: Tool Definitions With Scope and Role Requirements Now we define the tools themselves. Each tool declares what scopes and roles are required to invoke it. These declarations live alongside the tool code, not in a separate config file. Proximity reduces the chance of drift between a tool and its authorization requirements. Create src/tools/database.py: Python # Each tool is a handler with declared requirements. TOOL_REGISTRY: dict[str, tuple[list[str], list[str], callable]] = { "read_customer_record": ( ["database.read"], # required scopes [], # required roles read_customer_record, # handler ), "update_customer_plan": ( ["database.write"], [], update_customer_plan, ), "list_all_customers": ( ["admin.users.read"], ["admin"], # admin role required list_all_customers, ), } A developer with database.read scope can read customer records but cannot update plans. A contractor with no scopes gets blocked from everything. An admin with admin.users.read scope and the admin role can list all customers. The registry is the single source of truth for access control. The server enforces it at request time without consulting a database. Here is one tool handler showing resource-level constraint enforcement: Python async def read_customer_record(customer_id: int, *, _token=None) -> dict: # Optional: enforce per-resource constraints from the token. if _token and hasattr(_token, "raw_claims"): constraint = _token.raw_claims.get("resource_constraints", {}) allowed_id = constraint.get("customer_id") if allowed_id is not None and customer_id != allowed_id: raise PermissionError( f"Token scoped to customer {allowed_id}, " f"requested customer {customer_id}" ) record = _CUSTOMER_DB.get(customer_id) if record is None: raise ValueError(f"Customer {customer_id} not found") return record The resource_constraints claim in the token is what turns "this agent can read customer data" into "this agent can read customer 48291 for the next sixty seconds." It is the difference between scoping to a database table and scoping to a row. Part 4: The Tool Allow-List Gate System prompts are not security controls. A prompt injection can rewrite an agent's intent mid-session and convince it to call a tool it was never meant to access. The only reliable defense is an infrastructure-level allow-list that rejects unauthorized tool calls regardless of what the LLM decides. The allow-list is derived directly from the tool registry. Any tool not in the registry is unreachable: Python ALLOWED_TOOLS: set[str] = set(TOOL_REGISTRY.keys()) This set is checked before scope and role evaluation. A tool that is not in the registry cannot be called, period. A tool that is in the registry but requires scopes the token does not carry gets a 403. A tool that is in the registry and the token carries the right scopes goes through. The distinction between "tool not in allow-list" and "tool forbidden for this agent" matters for debugging and audit. The first indicates a misconfiguration or an attack. The second indicates a legitimate agent attempting an unauthorized operation, which itself is worth logging. Part 5: The Audit Logger Every tool call, successful or blocked, produces an audit log entry with the full delegation chain. The format is JSON Lines: one JSON object per line, ingestible by any SIEM, Splunk, or grep. Create src/audit/logger.py: Python class AuditLogger: def __init__(self, filepath: str | Path = "audit.log") -> None: self._path = Path(filepath) self._path.touch(exist_ok=True) def record(self, event: str, token: ValidatedToken, tool_name: str = "", tool_args: dict | None = None, result_summary: str = "", error: str = "") -> None: entry = { "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "event": event, "correlation_id": str(uuid.uuid4()), "subject": token.subject, "email": token.email, "roles": token.roles, "scopes": sorted(token.scopes), "delegation_chain": token.delegation_chain, "tool": tool_name, "tool_args": tool_args or {}, "result": result_summary, "error": error, } with open(self._path, "a") as fh: fh.write(json.dumps(entry, default=str) + "\n") When an auditor asks "who authorized this data access," the answer is in the log, not in a code review three weeks later. A correctly logged tool call looks like this: Python { "timestamp": "2026-06-14T14:04:00Z", "event": "tool_call", "subject": "alice-developer", "email": "[email protected]", "roles": ["developer"], "scopes": ["database.read", "email.send"], "delegation_chain": ["bob-admin"], "tool": "read_customer_record", "tool_args": {"customer_id": 1001}, "result": "ok" } Delegation chain flow: Human → Orchestrator Agent → Sub-Agent → MCP Server. The delegation chain reads: Bob (admin) delegated to Alice's developer agent, which called read_customer_record for customer 1001 at 14:04 UTC. If your logs cannot produce that sentence, your AI identity program is not operational. Part 6: Assembling the Server The main server wires together the token validator, the tool allow-list, the scope and role checks, the tool handlers, and the audit logger. Every request flows through them in order. Create src/server.py. Here is the core request path: Python token_validator = TokenValidator( jwks_url=OIDC_JWKS_URL, issuer=OIDC_ISSUER, audience=OIDC_AUDIENCE, clock_tolerance=30, ) audit = AuditLogger(AUDIT_LOG_FILE) async def mcp_tool_endpoint(request: Request) -> JSONResponse: # 1 — Extract and validate the Bearer token. auth = request.headers.get("Authorization", "") if not auth.startswith("Bearer "): raise AuthError("Missing Bearer token", 401) token_str = auth[7:] try: token = await token_validator.validate(token_str) except AuthError: audit.record("auth_failure", ...) raise # 2 — Parse the tool invocation. body = await request.json() tool_name = body.get("tool", body.get("name", "")) tool_args = body.get("arguments", body.get("args", {})) # 3 — Tool allow-list enforcement. if tool_name not in ALLOWED_TOOLS: audit.record("tool_allow_list_block", token, tool_name=tool_name) return JSONResponse( {"error": f"Tool '{tool_name}' is not authorized"}, status_code=403, ) # 4 — Scope + role authorization. required_scopes, required_roles = get_tool_requirements(tool_name) if required_scopes and not token.has_any_scope(required_scopes): return JSONResponse( {"error": "Insufficient scopes", "required": required_scopes, "granted": sorted(token.scopes)}, status_code=403, ) if required_roles: if not (set(token.roles) & set(required_roles)): return JSONResponse( {"error": "Insufficient role", "required_one_of": required_roles, "have": sorted(token.roles)}, status_code=403, ) # 5 — Execute and audit. handler = TOOL_REGISTRY[tool_name][2] result = await handler(**tool_args, _token=token) audit.record("tool_call", token, tool_name=tool_name, tool_args=tool_args, result_summary=str(result)[:200]) return JSONResponse({"result": result}) The 401 response format is specified by the MCP specification. The WWW-Authenticate header with resource_metadata is how clients discover that authentication is required: Python async def auth_error_handler(request, exc): return Response( content='{"error":"' + exc.args[0] + '"}', status_code=401, media_type="application/json", headers={ "WWW-Authenticate": ( f'Bearer resource_metadata=' f'"{AUDIENCE}/.well-known/oauth-protected-resource",' f'error="invalid_token"' ), }, ) Part 7: The Demo Agent To verify the server end-to-end without configuring a real OAuth provider, the companion project includes a demo agent that generates self-signed tokens for three simulated identities. Run it with python demo/agent.py --demo. The demo creates three agents with progressively restricted access: Plain Text Agent 1: Alice — developer, scopes: database.read + email.send ✓ Can read customer records ✗ Cannot update plans (missing database.write) ✗ Cannot list all customers (missing admin role) Agent 2: Bob — admin, scopes: database.read + database.write + admin.users.read ✓ Can read customer records ✓ Can update plans ✓ Can list all customers Agent 3: Carol — contractor, scopes: (none) ✗ Blocked from everything This is not a theoretical exercise. In the Stryker attack of March 2026, a compromised admin credential, one identity, over-privileged, with no scoping, allowed attackers to remotely wipe 200,000 devices across 79 countries. The attack did not use malware. It used the platform's own legitimate wipe functionality. The credential had no scope limiting it to a subset of devices, no short lifetime, and no audit trail that would have surfaced the anomaly before tens of thousands of endpoints were erased. Part 8: Testing The companion project includes a 13-test suite that verifies every security gate. Run it with: Python python -m pytest tests/ -v The test matrix covers the decision table exhaustively: TestConditionExpectedNo tokenMissing Authorization header401Invalid tokenMalformed JWT401Expired tokenexp in the past401Valid token + correct scopedatabase.read calling read_customer_record200Valid token + wrong scopeemail.send calling read_customer_record403Valid token + missing scopedatabase.read calling update_customer_plan403Valid token + correct scopesdatabase.read database.write calling update_customer_plan200Valid token + wrong roledeveloper role calling list_all_customers403Valid token + correct roleadmin role calling list_all_customers200Unknown tooldelete_everything not in allow-list403Discovery: protected resourceUnauthenticated GET200Discovery: authorization serverUnauthenticated GET200Audit log entriesTool call with delegation chainWritten with full chain Each test generates a real RSA key pair, signs a JWT with it, loads a matching JWKS, and sends a request through the full server stack using Starlette's TestClient. No mocking of the auth layer. The tests exercise the actual token validation code path. Part 9: Common Pitfalls localhost vs 127.0.0.1 redirect URI mismatch. MCP clients running locally often register 127.0.0.1 as their redirect URI, but the authorization server redirects to localhost (or vice versa). The Upstash OAuth deep-dive documents this as the most common integration failure. Normalize both addresses at registration and at token exchange. Cursor re-registers OAuth clients on every connection. The Dynamic Client Registration endpoint must handle the same client identity registering repeatedly. Store by client identity, not by registration request. Idempotency is critical. Clock skew causing spurious rejections. A 30-second clockTolerance is the pragmatic default. Distributed systems have clock drift. Rejecting a valid token because the IdP's clock is 12 seconds ahead of yours is a self-inflicted outage. Forgetting to serve discovery endpoints over HTTPS. MCP clients will refuse to fetch well-known URIs over plain HTTP in production. If your server is behind a load balancer, ensure the resource_url reflects the externally visible HTTPS URL, not the internal service name. Logging Bearer tokens. Sanitize the Authorization header from request logs. A leaked Bearer token in your logging pipeline is an identity compromise waiting to happen. The audit logger in this project intentionally records the validated identity, never the raw token. Production Hardening Before deploying to production, lock down the following: PKCE (S256) is mandatory. MCP clients are public clients without a client secret. PKCE is the only defense against authorization code interception.Short-lived tokens. Fifteen to sixty minutes, with refresh token rotation. Each use of a refresh token invalidates the previous one.HTTPS only. HTTP must be rejected at the network level. The MCP security best practices specification explicitly prohibits plaintext.Session-based authentication is prohibited. The MCP spec mandates token-based authentication. No cookies, no sessions.Audit log rotation and retention. JSON Lines accumulate quickly at production throughput. Configure log rotation and feed the audit stream to your SIEM. What We Built We built an MCP server that accepts tool calls only from authenticated agents. It validates OAuth 2.1 Bearer tokens using stateless JWKS-based validation with automatic key rotation. It enforces tool-level scopes and roles. A developer with database.read cannot write. A contractor with no scopes gets blocked from everything. An admin with the right role and scope can list all records. It maintains an infrastructure-level tool allow-list that rejects unauthorized tool calls regardless of what the LLM decides. It logs every access decision with the full delegation chain, so an auditor can trace any action back to the human who authorized it. The standards to do this at scale are maturing rapidly. SPIFFE handles workload identity. RFC 8693 covers token exchange with delegation chains. The IETF AIMS framework addresses agent identity. The engineering to do it in a single Python file is deployable today. The companion project is available on GitHub with setup instructions, a working demo, and a 13-test suite. Clone it, configure your OAuth provider, and you have an identity-aware MCP server in under 200 lines of application code. GitHub repository: github.com/pravin-khandke/identity-aware-mcp-server Clone it and run the demo in under two minutes: Shell git clone https://github.com/pravin-khandke/identity-aware-mcp-server.git cd identity-aware-mcp-server python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt python demo/agent.py --demo All code shown in this article is extracted from the repository. See src/auth/middleware.py for the JWKS validator, src/server.py for the full request pipeline, and tests/test_server.py for the 13-test suite.

The evolution from monolithic applications to microservices transformed enterprise software by decomposing business capabilities into independently deployable services. REST APIs, asynchronous messaging, and service discovery enabled systems that scaled both organizationally and technically. Although this model remains effective for deterministic business logic, the emergence of AI agents introduces a different execution paradigm. Instead of invoking predefined endpoints, an agent receives an objective, reasons about available capabilities, selects appropriate services, and dynamically composes a workflow. This shift changes service boundaries from business functionality to decision-making and capability orchestration. Why This Matters Traditional microservices assume that applications already know which services to invoke. An Order Service calls Inventory, Payment, and Shipping because the workflow is explicitly encoded during development. An AI agent, however, begins with an intent rather than an execution path. A request such as "purchase the least expensive laptop available and deliver it tomorrow" requires evaluating inventory, pricing, promotions, shipping constraints, and fraud policies before any API is called. The workflow is determined during execution instead of implementation. A conventional orchestration service typically resembles the following implementation. Java public OrderResponse checkout(OrderRequest request) { Inventory inventory = inventoryClient.reserve(request); Payment payment = paymentClient.authorize(request); Shipping shipment = shippingClient.schedule(request); return new OrderResponse(payment, shipment); } The implementation is deterministic because every dependency is known beforehand. Adding another payment gateway or shipping provider requires modifying orchestration logic, gradually increasing coupling between services. As enterprises integrate AI-driven workflows, continuously extending predefined execution paths becomes increasingly difficult. Agent Services replace hardcoded dependencies with capability discovery. Rather than directly invoking an Inventory Service, the runtime identifies which registered capability satisfies the current intent. Java public Tool resolve(Intent intent) { return toolRegistry.stream() .filter(tool -> tool.supports(intent)) .findFirst() .orElseThrow(() -> new ToolNotFoundException(intent.name())); } The registry enables services to advertise capabilities instead of exposing only procedural APIs. Existing microservices remain responsible for inventory reservation, payment authorization, or shipment scheduling, but the responsibility for deciding which capability should execute moves into an intelligent coordination layer. New business capabilities can therefore be introduced without rewriting orchestration code. This distinction fundamentally changes API design. Traditional REST endpoints expose operations such as /reserveInventory or /authorizePayment. Agent-oriented systems instead expose semantic capabilities like "find lowest cost supplier," "recommend shipping option," or "detect payment risk." These descriptions allow planning engines to reason about business objectives instead of matching endpoint names. Reasoning requires an additional architectural component capable of translating natural language into executable plans. This responsibility belongs to an Intent Router, which functions similarly to an API Gateway but routes requests based on semantic meaning rather than URLs. Java public ExecutionPlan plan(String goal) { Intent intent = classifier.classify(goal); Tool tool = registry.resolve(intent); return planner.create(tool, goal); } The classifier converts an objective into structured intent, the registry discovers an appropriate capability, and the planner generates an execution strategy. Once planning completes, downstream execution remains deterministic. Large language models participate only during reasoning, while conventional microservices continue enforcing validation rules, transactional consistency, and domain constraints. Separating planning from execution preserves enterprise reliability while introducing adaptive behavior. This separation also dispels a common misconception that AI agents replace microservices. Business logic continues to belong inside deterministic services because payment authorization, inventory consistency, pricing calculations, and compliance rules require predictable execution. Agent Services instead provide an intelligent layer responsible for selecting, coordinating, and sequencing those services according to business objectives. Rather than replacing existing architectures, they extend them with decision-making capabilities that previously existed only inside application code. Consequently, service boundaries begin shifting away from business entities toward reusable decision engines. Instead of embedding procurement, logistics, or fraud decisions inside multiple applications, organizations can expose these responsibilities as independent Agent Services that orchestrate existing microservices. The underlying APIs remain stable while reasoning evolves independently, enabling enterprise systems to become progressively more adaptive without sacrificing the deterministic foundations that made microservice architectures successful. Taking Memory Into Account Memory becomes the next architectural concern once planning is separated from execution. Stateless REST requests work well for isolated transactions, but agents frequently solve objectives through multiple reasoning cycles. Intermediate decisions, retrieved knowledge, user preferences, and execution history must persist beyond a single request. This context is operational rather than transactional. Business entities continue residing in relational databases, while the agent memory layer preserves reasoning state that enables future decisions to remain consistent. Java public AgentContext update(String sessionId, Observation observation) { AgentContext context = repository.load(sessionId); context.append(observation); repository.save(context); return context; } Rather than storing business records, the memory layer continuously enriches execution context with observations generated during planning. Future reasoning cycles consume this accumulated context instead of repeatedly querying downstream services, reducing redundant tool execution while maintaining continuity across long-running workflows. As objectives become more sophisticated, a single agent rarely owns every required capability. Instead of directly invoking multiple APIs, an agent can delegate specialized responsibilities to another agent while maintaining overall coordination. This interaction is based on expertise rather than ownership, allowing procurement, logistics, compliance, or fraud agents to evolve independently while sharing the same underlying microservices. Java AgentResponse response = logisticsAgent.execute( new AgentTask( "Optimize shipping route", context)); Delegation transfers structured objectives instead of procedural API calls. Each agent independently plans its assigned task before returning a deterministic result. Existing Inventory, Payment, and Shipping services remain unchanged, while the coordination layer becomes modular and extensible. Observability Implications Observability must also evolve because traditional distributed tracing explains service execution but not decision making. Understanding why an agent selected one capability over another is equally important as measuring latency or availability. Reasoning traces therefore become first-class telemetry alongside conventional application metrics. Java Span span = tracer.nextSpan() .name("agent.plan"); span.tag("goal", goal); span.tag("selectedTool", tool.name()); span.tag("confidence", score.toString()); span.end(); Capturing planning metadata allows engineering teams to correlate business outcomes with reasoning quality. An operation may succeed technically while producing an incorrect recommendation because the planner selected an unsuitable capability. Monitoring therefore expands beyond response times to include tool selection, planning confidence, execution cost, and reasoning latency. Autonomous planning also introduces governance challenges. Traditional services authorize callers before executing business logic, whereas Agent Services must additionally validate that planners invoke only approved capabilities. Every tool should expose explicit permissions and execution policies so that reasoning engines remain constrained by enterprise governance regardless of how plans are generated. Java public ToolResult execute(AgentTask task) { policyEngine.authorize(task.agent(), task.tool()); return toolExecutor.run(task); } Separating authorization from planning ensures deterministic policy enforcement around probabilistic reasoning. Existing identity providers, audit systems, and compliance frameworks remain applicable because execution ultimately flows through governed business capabilities rather than unrestricted model outputs. A Final Word The transition from microservices to Agent Services is therefore not a replacement of proven architectural principles but their natural evolution. Microservices continue delivering transactional consistency, persistence, and deterministic business logic, while Agent Services introduce planning, semantic routing, capability discovery, memory, and adaptive orchestration. The architectural boundary shifts from exposing operations to exposing decisions, allowing intelligent planners to compose existing services according to business objectives rather than predefined workflows. Enterprise platforms adopting this layered approach preserve the reliability of mature microservice ecosystems while gaining the flexibility required for AI-native applications, making Agent Services the next logical abstraction for software systems where reasoning becomes as important as execution.

JSON has been the default structured-data format for APIs, configuration, event streams, and application integration for decades. It is portable, readable, widely supported, and easy to validate. However, JSON was not designed for LLMs. When structured data is placed inside an LLM prompt, every quotation mark, repeated field name, brace, comma, and nested structure contributes to the prompt’s token count. For a small request, this overhead may be insignificant. For applications that send thousands of records, tool results, or knowledge-graph entities to an LLM, it can consume a meaningful portion of the context window. Token-Oriented Object Notation, or TOON, proposes a different representation. It encodes the same objects, arrays, and primitive values as JSON but uses a compact, line-oriented syntax designed for LLM prompts. TOON combines indentation for nested structures with tabular representations for homogeneous arrays. Its strongest use case is a collection of objects that share the same fields. TOON-LD applies a related idea to Linked Data. It is intended to represent JSON-LD knowledge graphs more compactly while retaining Linked Data constructs such as @context, @id, @type and @graph. This tutorial explains the differences among JSON, TOON, JSON-LD & TOON-LD and shows how to benchmark their token consumption, serialized size, conversion overhead and round-trip correctness. Why JSON Consumes Additional LLM Tokens Consider the following incident records: JSON { "incidents": [ { "id": "INC-000001", "service": "checkout", "severity": "critical", "region": "ap-south-1", "owner": "platform" }, { "id": "INC-000002", "service": "payments", "severity": "high", "region": "eu-west-1", "owner": "payments" } ] } The field names id, service, severity, region, and owner appear in every record. An application parser needs those repeated keys to reconstruct each JSON object, but an LLM prompt pays for their repeated tokenization. A corresponding TOON representation can declare the fields once and place the values in rows: Plain Text incidents[2]{id,service,severity,region,owner}: INC-000001,checkout,critical,ap-south-1,platform INC-000002,payments,high,eu-west-1,payments The exact encoded output depends on the TOON specification and encoder version, so production applications should generate TOON through a library rather than manually constructing it. The important difference is structural: JSON repeats the complete object syntax for every row, whereas TOON can amortize that structure across a uniform collection. The TOON project describes the format as a lossless representation of the JSON data model and identifies uniform arrays of objects as its primary efficiency advantage. It also notes that deeply nested or non-uniform data may not receive the same benefit and can sometimes remain more efficient in JSON. JSON and TOON Serve Different Architectural Purposes TOON should not automatically replace JSON across an application. JSON remains appropriate for: Public and internal APIsApplication configurationPersistent storageEvent exchangeSchema-based validationBrowser and programming-language interoperabilityObservability logs and audit records TOON is better evaluated as a representation used at the LLM boundary. A practical architecture is: The application continues to use JSON internally. Only the structured context inserted into the prompt is converted to TOON. This approach reduces migration risk and confines the new format to the part of the architecture where token efficiency matters. What Is JSON-LD? JSON-LD is a W3C-standardized JSON-based format for Linked Data. It adds semantic meaning to ordinary JSON through globally identifiable concepts and relationships. The JSON-LD 1.1 specification is a W3C Recommendation and is designed to integrate Linked Data into JSON-based programming environments and web services. Consider the following example: JSON-LD { "@context": { "ex": "https://example.org/", "affects": { "@id": "ex:affects", "@type": "@id" }, "ownedBy": { "@id": "ex:ownedBy", "@type": "@id" } }, "@graph": [ { "@id": "ex:incident-101", "@type": "ex:Incident", "ex:severity": "critical", "affects": "ex:checkout" }, { "@id": "ex:checkout", "@type": "ex:Service", "ownedBy": "ex:platform-team" } ] } This document contains more than two nested JSON objects. It describes a graph: An LLM can use this structure for questions such as: Which team owns the service affected by incident 101? JSON-LD is therefore useful for knowledge graphs, semantic search, Graph-RAG, interoperable metadata, and agent systems that must traverse relationships among entities. What Is TOON-LD? TOON-LD is an emerging format that extends TOON with Linked Data semantics. Its implementation describes TOON-LD as a compression representation for JSON-LD knowledge graphs used in LLM context windows. It supports JSON-LD constructs and provides conversions between JSON-LD and TOON-LD. A simplified TOON-LD representation of a uniform graph may resemble: Plain Text @context: ex: https://example.org/ @graph[2]{@id,@type,ex:severity,ex:affects}: ex:incident-101,ex:Incident,critical,ex:checkout ex:incident-102,ex:Incident,high,ex:payments The main optimization again comes from declaring a common shape once instead of repeating every JSON-LD field for every entity. TOON-LD should nevertheless be assessed differently from JSON-LD. JSON-LD is a mature W3C standard with established processors and semantic-web tooling. TOON-LD is considerably newer and should be evaluated for library stability, interoperability, and semantic preservation before production use. JSON, TOON, JSON-LD and TOON-LD Compared Format Data model Main objective Typical use JSON Object and array tree Universal structured-data exchange APIs, events, configuration and storage TOON JSON-compatible object and array tree Reduce tokens in LLM context Prompt records, RAG context and tool results JSON-LD RDF-compatible linked graph Semantically interoperable Linked Data Knowledge graphs and semantic metadata TOON-LD Token-oriented linked graph Reduce JSON-LD context tokens Graph-RAG and knowledge-driven agents TOON should be compared with JSON. TOON-LD should primarily be compared with JSON-LD. Comparing TOON-LD only with ordinary JSON would mix two different data models and could produce a misleading conclusion. Designing a Fair Benchmark Token-efficiency claims should not be evaluated with one carefully selected payload. The accompanying benchmark uses four datasets: Flat homogeneous incident recordsNested homogeneous incident recordsIrregular and sparse incident recordsJSON-LD incident knowledge graphs Each dataset is generated at multiple scales: 10 records,100 records, 1000 records, 10000 records This exposes an important characteristic of token-oriented formats: their benefits can depend significantly on the shape and scale of the input. Flat Homogeneous Data The flat dataset contains records with identical fields: JSON { "id": "INC-000001", "service": "service-01", "severity": "critical", "region": "ap-south-1", "owner": "platform", "latency_ms": 450, "retryable": true } This is likely to be the strongest scenario for TOON because the schema can be declared once and reused for all rows. Nested Data The nested dataset includes workload, metric, and status objects: JSON { "id": "INC-000001", "workload": { "namespace": "team-1", "deployment": "service-01", "pod": "service-01-000001" }, "metrics": { "cpu_percent": 72, "memory_mib": 850, "latency_ms": 450 }, "status": { "severity": "critical", "acknowledged": false } } This tests whether TOON’s reduced punctuation compensates for indentation and nested structural markers. Irregular Data The irregular dataset intentionally varies fields across records: JSON [ { "id": "INC-000001", "service": "checkout", "severity": "critical" }, { "id": "INC-000002", "dependencies": ["postgresql", "kafka"], "retry_after_seconds": 30 }, { "id": "INC-000003", "error": { "code": 503, "message": "upstream unavailable" } } ] This is important because tabular formats perform best when records share a schema. Sparse or heterogeneous structures can reduce or eliminate that advantage. Linked-Data Graph The final dataset contains incidents, services, teams, and relationships expressed through JSON-LD. This evaluates TOON-LD against the representation it is intended to optimize. Metrics Used in the Experiment The benchmark records the following metrics. Serialized Characters This is the number of Unicode characters in the encoded document. Character count is easy to understand, but it is not a substitute for token count. Different tokenizers divide the same text differently. UTF-8 Bytes The benchmark measures the encoded byte length using: len(serialized_value.encode("utf-8")). This helps estimate storage and network-transfer overhead. Token Count Token count is measured using the selected tokenizer. The repository defaults to the o200k_base tokenizer but allows another tokenizer to be configured. For linked data, JSON-LD replaces JSON in the calculation. Token savings are tokenizer-specific. A result measured with one tokenizer should not be presented as universally applicable to every model family. Encoding Latency Encoding latency measures the time required to convert an in-memory object to JSON, TOON, JSON-LD, or TOON-LD. The benchmark reports: median encoding latency;95th-percentile encoding latency. Decoding Latency Decoding latency measures the time required to reconstruct the application data from its serialized representation. This matters because reducing prompt tokens may introduce additional CPU overhead in the application. Peak Memory Python’s tracemalloc module records the peak memory observed during serialization. Round-Trip Correctness For every measured iteration, the benchmark verifies: source data == decode(encode(source data)) A format that produces a smaller prompt but cannot reliably reconstruct the source data is unsuitable for lossless interchange. Running the Benchmark Clone the repository: Shell git clone https://github.com/jojustin/json-toon-toonld-benchmark.git cd json-toon-toonld-benchmark Create a virtual environment: Shell python -m venv .venv source .venv/bin/activate Install the dependencies: Shell pip install -r requirements.txt Run a small validation experiment first: Shell python -m src.run_benchmark --sizes 10 100 --iterations 5 Run the complete benchmark: Shell python -m src.run_benchmark --sizes 10 100 1000 10000 --iterations 30 To calculate percentage reductions and encoding overhead: Shell python -m src.summarize Run the automated tests: Shell pytest -q Why the Benchmark Uses Minified JSON A TOON comparison can be exaggerated by comparing it only with pretty-printed JSON. Pretty-printed JSON contains indentation and line breaks intended for human readability: JSON { "id": 1, "name": "Alice" } Minified JSON removes optional whitespace: JSON {"id":1,"name":"Alice"} Since production systems can easily minify JSON before placing it in a prompt, minified JSON is the appropriate primary baseline. Pretty-printed JSON can still be reported as a separate readability baseline, but it should not be the only comparison. Interpreting the Expected Results The benchmark results show that token-oriented serialization is not uniformly more efficient than JSON. Its effectiveness depends strongly on the structure of the input data. TOON performs best when the input consists of flat, homogeneous records that share the same fields, while compact JSON remains more efficient for irregular and deeply nested structures. TOON and TOON-LD also introduce measurable conversion overhead because their encoders must analyze the input structure and generate a more specialized representation. Token Efficiency For the flat dataset, TOON reduced the token count from approximately 39,500 tokens to 23,000 tokens, corresponding to a reduction of about 42%. This result represents TOON’s intended use case: a large collection of records sharing a common schema. Rather than repeating every field name for each record, TOON declares the fields once and represents the values in a tabular form. The result was different for irregular data. Compact JSON required approximately 27,300 tokens, while TOON required about 33,000 tokens — an increase of approximately 21%. Because the records contained different fields and structures, TOON could not efficiently amortize a shared schema across the collection. The additional structural notation therefore outweighed the savings obtained by removing JSON punctuation. A similar pattern appeared in the nested dataset. TOON used approximately 74,000 tokens compared with 64,000 tokens for compact JSON, representing an increase of around 16%. The result indicates that deeply nested objects are not necessarily well suited to tabular token-oriented encoding. Indentation, nested object markers, and repeated hierarchical structures can make TOON less compact than minified JSON. For the linked-data dataset, TOON-LD reduced the representation from approximately 40,000 JSON-LD tokens to 28,500 tokens, a saving of about 29%. This demonstrates the potential of schema-aware linked-data compression. However, the token reduction must be interpreted together with the round-trip validation results. In the tested implementation, the reconstructed TOON-LD output did not preserve valid JSON-LD semantics. The observed token saving therefore represents compression potential, but not a verified lossless transformation for this workload. Encoding Performance JSON consistently encoded faster than TOON. For the flat dataset, compact JSON required approximately 6 milliseconds, whereas TOON required around 27 milliseconds. TOON was therefore about four times slower, despite producing a substantially smaller token representation. The irregular dataset showed a similar pattern. JSON encoding took approximately 5 milliseconds, while TOON required nearly 30 milliseconds. In this case, TOON introduced significant processing overhead while also producing more tokens, making compact JSON preferable on both efficiency and runtime grounds. For the nested dataset, JSON required approximately 12 milliseconds and TOON approximately 55 milliseconds. This was the highest TOON encoding time observed among the datasets. The additional processing required to traverse and represent deeply nested structures contributed to both higher runtime and higher token count. JSON-LD encoding required approximately 6 milliseconds for the linked-data dataset, compared with about 17 milliseconds for TOON-LD. TOON-LD was therefore around three times slower to encode, although its absolute processing time remained below 20 milliseconds for 1,000 records. These results show that reduced token count is not computationally free. TOON and TOON-LD shift some work from the LLM prompt to the application’s serialization layer. End-to-End Conversion Overhead For flat data, TOON introduced approximately 29.14 milliseconds of additional conversion time compared with JSON. For irregular data, the overhead increased to 31.94 milliseconds. The linked-data comparison produced the lowest overhead: TOON-LD added approximately 11.59 milliseconds relative to JSON-LD. The nested dataset generated the largest conversion overhead at 58.93 milliseconds. This finding is consistent with the encoding-time and token-count results: nested structures were both slower to process and less token-efficient in TOON. Although these overheads are small compared with the end-to-end latency of many remote LLM requests, they may still matter in high-throughput systems, local inference pipelines, or workflows that repeatedly serialize and deserialize large payloads. Conversion cost should therefore be evaluated relative to the expected inference savings and request volume. Overall Interpretation The combined results reveal three distinct workload categories. Workload Token outcome Conversion outcome Recommendation Flat, homogeneous records About 42% fewer tokens About 29 ms additional conversion time Strong candidate for TOON Irregular records About 21% more tokens About 32 ms additional conversion time Prefer compact JSON Deeply nested records About 16% more tokens About 59 ms additional conversion time Prefer compact JSON Linked data About 29% fewer tokens About 12 ms additional conversion time Promising, but semantic validation must pass The strongest result is that data shape is the primary determinant of TOON efficiency. TOON is effective for uniform, tabular collections because it avoids repeating field names. It is less suitable for sparse, irregular, or deeply nested data, where compact JSON can require fewer tokens and substantially less conversion time. The linked-data result should be treated cautiously. Although TOON-LD reduced token usage and introduced relatively modest conversion overhead, the tested implementation failed semantic round-trip validation. It should therefore not be presented as a lossless JSON-LD replacement for this experiment. A practical selection policy derived from the results is: Flat and homogeneous records → TOON Irregular or nested records → Compact JSON Linked-data graphs → JSON-LD unless TOON-LD semantic validation passes Overall, the benchmark supports using TOON as a selective prompt-boundary optimization, rather than as a universal replacement for JSON. The appropriate decision should consider token reduction, conversion overhead, structural correctness, and semantic preservation together. Extending the Benchmark With LLM accuracy The repository focuses on deterministic, provider-neutral measurements. A second experiment can assess how well an LLM understands each representation. Use semantically identical questions for JSON and TOON: List the IDs of all critical incidents owned by the platform team. Return only a JSON array of incident IDs. For JSON-LD and TOON-LD, include multi-hop questions: Which teams own services affected by critical incidents? Measure: Input tokensOutput tokensTime to first tokenTotal response latencyExact-match accuracyPrecision, recall, and F1Invalid-output rateHallucination rateCost per request Keep these variables constant: Model and model versionSystem promptQuestionTemperatureMaximum output tokensDatasetNumber of repeated trials Randomize the order of JSON and TOON trials so that temporary service conditions do not consistently favor one format. When Should TOON Be Considered? TOON is worth evaluating when: Large homogeneous datasets are repeatedly placed in promptsPrompt-token cost is significantContext-window capacity is constrainedThe application controls both encoding and decodingStructured context is primarily read by the modelBenchmarked accuracy remains acceptable TOON may be less attractive when: Payloads are smallObjects are deeply nested or highly irregularStandard interoperability is more important than token savingsThe model must reliably generate complex TOON outputDownstream tools require JSON directlyConversion complexity exceeds measurable savings When Should TOON-LD Be Considered? TOON-LD may be useful when: A Graph-RAG pipeline inserts many JSON-LD entities into promptsRepeated graph entities share common shapesA semantic agent receives linked relationships as contextPreserving @context, identifiers, and graph relationships is essentialJSON-LD token consumption limits useful graph size It should be approached cautiously when: External systems expect standards-compliant JSON-LD directlyRDF canonicalization and semantic round trips have not been testedPackage maturity and long-term compatibility are criticalThe linked-data graph contains complex or highly heterogeneous structures Security Considerations Structured-data compression does not eliminate prompt-security concerns. Before inserting TOON or TOON-LD content into a prompt: Treat serialized values as untrusted dataSeparate instructions from retrieved contentValidate decoded responsesEnforce output schemas where possibleLimit graph traversal and retrieved entity countsPrevent untrusted content from altering system instructionsLog the canonical JSON or JSON-LD source for auditability For TOON-LD, external contexts and linked identifiers should also be controlled. Applications should avoid dereferencing arbitrary remote contexts or URLs without appropriate allowlists, timeouts and content validation. Conclusion JSON remains the correct default for general-purpose application integration. It has unmatched interoperability, mature tooling, schema support and broad developer familiarity. TOON addresses a narrower problem: reducing the token overhead of structured data passed to language models. Its strongest potential advantage is in large, homogeneous collections where repeated JSON keys consume substantial context. TOON-LD applies the same general principle to JSON-LD knowledge graphs. It may allow Graph-RAG and semantic-agent systems to place more linked data in an LLM context, but it is newer and requires careful testing for semantic equivalence and implementation maturity. The key decision should not be based on token reduction alone. A production evaluation should measure: Token countSerialized bytesEncoding and decoding overheadMemory usageRound-trip correctnessLLM comprehensionStructured-output reliabilityEnd-to-end latencyCost at realistic request volumes A practical adoption pattern is to retain JSON or JSON-LD as the canonical application representation and introduce TOON or TOON-LD only as an explicitly measured prompt-boundary optimization. The accompanying benchmark provides a reproducible starting point for making that decision with evidence rather than assumptions.

By Josephine Eskaline Joyce CORE

On one of our projects, we were building microfrontends, and at some point we wanted to add SSR. The reasons were the usual ones: better first paint, fewer layout shifts, real content for crawlers, less JS to load before something appears on screen. Setting it up turned out to be harder than I expected. There was no obvious out-of-box path that fit our setup, and most of the approaches I found either assumed a shared build or asked us to add new infrastructure on top of what we already had. That is what made me start sketching a small package. Something any team could drop in and get SSR for their microfrontend without rewriting either side. The result is @mf-toolkit/mf-ssr. The rest of this is about the approach behind it, since I think that is the interesting part. What I Wanted I started from a short list, taken straight from how I'd want to use such a thing: MF content on first paint. The remote's HTML should arrive inside the host's server response, not be fetched from the client after JS loads. No empty slot, no layout shift, real content in crawlers.No shared build, no central orchestrator. Each team builds and deploys their remote on their own schedule. The host should not need a special Node process that imports every remote into one bundle, and remote teams should not need to rewrite their bundler config to fit a central setup.Two paths for two setups, one host component. I wanted both scenarios covered. url mode for when the remote team runs their own server and wants to own SSR on their side (and possibly use a non-React framework). loader mode for when the remote only ships a static React bundle and the host server can do the SSR for it. The host code should look almost the same in either case, with just a single prop telling the component which path to use.Any framework, any runtime. The remote might be React, but it could be Vue, Svelte, or anything else. The host shouldn't care. And on the server, the same code should run on Node, Bun, Cloudflare Workers, or Vercel Edge with no rewrites.Host state still drives the remote after hydration. When the host re-renders with new props, the remote should re-render too. No re-fetch, no re-mount, no shared store between bundles.Honest failure modes. A timeout when the remote is slow, retry when a request fails, an explicit fallback for total failure, and a cache that respects auth boundaries. The things that decide whether SSR is a win or a regression when one team has a bad deploy. The last bullet is what most articles skip. SSR is easy in the happy path. The interesting code is what happens when one of the remotes is slow, down, or returning garbage. How It Works The idea is small: Instead of importing remote components into the host server, the host pulls the rendered output in over HTTP at SSR time and streams it into its own response. The browser gets a full page on first paint. How that "pull" happens depends on how the remote is deployed. The package supports two modes for that: url mode – the remote has its own HTTP endpoint that returns rendered HTML. The host fetches that HTML during SSR.loader mode – the remote is a static React bundle on a CDN or S3, no server behind it. The host imports the component directly during SSR and renders it inline. Same host component (<MFBridgeSSR>) in both cases, just one prop changes. Both modes can live on the same page. The interesting part is what happens after hydration. The host has to push prop changes into the remote without re-fetching anything. I will get to that in a moment. I'll start with url mode since it is the more general case (any framework on the remote side, any runtime on the server), and then cover loader mode separately. url mode: Remote With Its Own HTTP Endpoint In url mode, the remote server does the SSR. The remote team runs their own runtime (Node, Bun, a Cloudflare Worker, a Next.js Route Handler, whatever they prefer) and exposes an HTTP endpoint that returns rendered HTML for the given props. The host's SSR pass just calls that endpoint and inlines the response into the page. Each microfrontend owns its own rendering pipeline. Remote Handler TypeScript-JSX import { createMFReactFragment } from '@mf-toolkit/mf-ssr/fragment' import { CheckoutWidget } from './CheckoutWidget' export const handler = createMFReactFragment(CheckoutWidget) handler is a plain Web fetch handler: (req: Request) => Promise<Response>. It reads props from the query string, renders the component to a stream with renderToReadableStream, and writes the props into a small <script> tag so the client can hydrate without going back to the network. One nuance worth flagging: those props go inside a <script> tag, so a raw </script> inside a string prop would close the tag prematurely and let user-controlled values escape into the HTML context. The handler escapes <, >, &, and U+2028/U+2029 to their \uXXXX equivalents before embedding. JSON.parse on the client treats them the same as the originals, but the browser's HTML parser never sees a closing tag. It is a few lines of code that close a real XSS hole. You wire the handler into whatever HTTP framework the remote team already uses. Hono, a Next.js Route Handler, Bun, plain Node, a Cloudflare Worker. The handler doesn't know about any of them. And because the whole thing is Web Streams, it runs on Cloudflare Workers, Vercel Edge, Bun, and Node 18+ without changes. Non-React Remotes createMFReactFragment is a React-only helper. If the remote is Vue, Svelte, Solid, or vanilla JS, the team writes their own fetch handler instead, but it has to produce the same HTML shape the host expects: TypeScript-JSX <div data-mf-ssr="checkout"> <script type="application/json" data-mf-props>{"orderId":"42"}</script> <div data-mf-app><!-- Vue / Svelte / whatever rendered HTML --></div> </div> The team uses their framework's SSR renderer (renderToString for Vue, Svelte's SSR API, and so on) to produce the inner HTML, and serializes props into the <script data-mf-props> tag, applying the same < / > / & escaping. On the client, the remote mounts itself into [data-mf-app] and reads initial props from [data-mf-props]. If it needs prop updates from the host after hydration, it listens on the same DOMEventBus (exported from @mf-toolkit/mf-bridge). The bus is a thin wrapper over native CustomEvent, with no React dependency, so it works fine for any framework. This path is more work than createMFReactFragment, but the contract is small and explicit. The host doesn't care which framework produced the inner HTML — as long as the wrapper structure matches, hydration finds the right slots. Host Component TypeScript-JSX <MFBridgeSSR url="https://checkout.acme.com/fragment" namespace="checkout" props={{ orderId, step } fallback={<CheckoutSkeleton />} /> During SSR, the host fetches the remote's HTML and streams it into the response. Each <MFBridgeSSR> lives in its own Suspense boundary, so a slow checkout doesn't block the header. They stream as they resolve. On the client, the host hydrates, then waits for prop changes coming from React. Prop Updates After Hydration This was the part I cared about most. The remote is in its own React root, often in its own bundle, sometimes in a completely different framework. You can't re-render it like a normal child. So I used the one thing both sides already share at runtime: the DOM node the remote is mounted into. When the host re-renders with new props, the host fires a CustomEvent on that node. The remote listens for it and re-renders its root with the new props. No re-fetch, no global state, no coupling between bundles beyond a shared namespace string. TypeScript-JSX // remote client entry import { hydrateWithBridge } from '@mf-toolkit/mf-bridge/hydrate' import { CheckoutWidget } from './CheckoutWidget' hydrateWithBridge(CheckoutWidget, { namespace: 'checkout' }) I picked this because it is isolated by construction. If a page has several MF slots, each one has its own mount node, so events never leak between them. And it is just DOM, so there is no bundler magic to debug when something goes wrong. Events and Commands Prop streaming is one direction. For the other direction, the same bus works in reverse. The host passes onEvent to receive events the remote emits, and a commandRef it can use to send imperative commands back: TypeScript-JSX const resetRef = useRef<((type: string, payload?: unknown) => void) | null>(null) <MFBridgeSSR url="https://checkout.acme.com/fragment" namespace="checkout" props={{ orderId } onEvent={(type, payload) => { if (type === 'orderPlaced') navigate('/thanks') } commandRef={resetRef} /> // somewhere in host code, e.g. when the user switches accounts: resetRef.current?.('reset') On the remote, hydrateWithBridge accepts an onCommand handler, and DOMEventBus (exported from @mf-toolkit/mf-bridge) lets the remote send events back: TypeScript-JSX import { hydrateWithBridge } from '@mf-toolkit/mf-bridge/hydrate' import { DOMEventBus } from '@mf-toolkit/mf-bridge' hydrateWithBridge(CheckoutWidget, { namespace: 'checkout', onCommand: (type) => { if (type === 'reset') store.reset() }, }) // inside the widget, after a successful payment: const container = document.querySelector<HTMLElement>('[data-mf-namespace="checkout"]')! new DOMEventBus(container, 'checkout').send('event', { type: 'orderPlaced', payload: { orderId }, }) The channel is the same DOMEventBus, just with extra event names on top of propsChanged. So everything I said earlier about isolation still holds: events on one slot don't reach another, even when the remote is the same. loader mode: Remote as a Static Bundle In loader mode, the host server does the SSR for the remote. The remote team ships only a static React bundle (CDN, S3, or a Module Federation host) and runs no server of their own. When the host renders its page server-side, it imports the remote component and renders it inline, the same way it renders any other component in the host tree. The remote has no SSR runtime and no rendering responsibility; the host does all the work.ё Host Component JSX const loadCheckout = () => import('checkout/Widget').then(m => m.CheckoutWidget) <MFBridgeSSR loader={loadCheckout} props={{ orderId, step } fallback={<CheckoutSkeleton />} /> That is everything. No namespace, no errorFallback tricks needed for hydration, no client entry to write on the remote side. The package wraps the loader in React.lazy and renders the component inside the host's React tree, both server-side and after hydration. Props, Events, Commands Since the remote lives inside the host's React tree, every kind of communication is just React: Props – re-render normally. When the host's parent component re-renders with new props, the remote re-renders too. No DOMEventBus, no hydrateWithBridge, no propsChanged events.Events from remote to host – pass a callback through props. The remote calls it like any other handler.Commands from host to remote – pass them through props as well, or expose a ref through forwardRef. If you find yourself wanting onEvent / commandRef here, you are probably reaching for url mode. Requirements A few constraints come with this mode: Host must be able to resolve the loader on the server. The package calls your loader() function as-is. It doesn't fetch bundles from URLs itself. In practice, this means Module Federation runtime on the host (or some other server-side dynamic import mechanism that knows how to find checkout/Widget). Without that, the import fails in Node before any rendering happens.React only. The host literally calls the component during SSR, so the remote has to be a React component. For Vue/Svelte/vanilla remotes, use url mode.SSR-safe import. The remote's exposed module has to be importable on the server, which means no window, document, or other browser globals at the module top level. Move that code inside useEffect or behind a typeof window check.Stable loader reference. Define loadCheckout at module scope or wrap it in useCallback. The package caches the resulting React.lazy by loader reference so Suspense retries reuse the same promise. A new function on every render would break that and trigger an infinite retry loop. When to Pick Which CategoryURL modeLoader modeRemote infrastructureOwn HTTP endpoint: Node.js, Bun, Worker, etc.Static bundle on CDN, S3, or Module Federation hostRemote frameworkAny: React, Vue, Svelte, vanilla JavaScriptReact onlyIsolationSeparate React root inside the remote bundleRendered inline in the host React treeProp updatesDOM events through DOMEventBusNative React re-renderEvents and commandsonEvent and commandRefReact props and refsBest forIndependent teams, mixed frameworks, and polyreposSimple React remotes with no extra infrastructure Both modes use the same <MFBridgeSSR> and can be mixed freely on the same page. The Corner Cases I Spent Time On A few production scenarios I wanted to make sure the package handled honestly. Graceful Degradation When the Remote Is Down A remote can be slow, return a 5xx, or simply not respond. The host page shouldn't break because of one bad slot. mf-ssr accepts an errorFallback, and the trick is that the fallback can be the same remote mounted on the client through mf-bridge: TypeScript-JSX import { MFBridgeSSR } from '@mf-toolkit/mf-ssr' import { MFBridgeLazy } from '@mf-toolkit/mf-bridge' <MFBridgeSSR url="https://checkout.acme.com/fragment" namespace="checkout" props={{ orderId } timeout={2000} errorFallback={ <MFBridgeLazy register={() => import('checkout/entry').then(m => m.register)} props={{ orderId } fallback={<CheckoutSkeleton />} /> } /> If the SSR fetch times out, the user still gets the widget. Just on the client, the same way it would have worked without mf-ssr at all. The page doesn't break. The slot loses its first-paint optimization, for that one request. When the remote recovers, the next render uses SSR again with no code change on either side. I like this case because it inverts the usual SSR-or-nothing tradeoff. SSR becomes the fast path, with a working client-side path sitting right behind it. Auth-Isolated Caching The host caches fragments by url + props + timeout. Fine for public content. Not fine when each user gets different HTML — they would share a cache slot and see each other's pages. So there is a cacheKey prop you set when the request carries auth: TypeScript-JSX <MFBridgeSSR url="https://account.acme.com/fragment" namespace="account" props={{ view: 'orders' } fetchOptions={{ headers: { authorization: `Bearer ${token}` } } cacheKey={userId} /> The other side of the same coin is public fragments. The remote's fragment endpoint accepts a cacheControl option, so you can serve a product card as public, s-maxage=60, stale-while-revalidate=30 and let a CDN cache it for everyone: TypeScript-JSX export const handler = createMFReactFragment(ProductCard, { cacheControl: 'public, s-maxage=60, stale-while-revalidate=30', vary: 'Accept-Language', }) One pattern handles per-user fragments, the other handles cacheable public ones. Same component on both sides. Multiple Instances of the Same Remote Header, sidebar, and a content slot can all be the same remote on one page. The reason I sent prop updates through the mount DOM node, instead of a global event bus, is exactly this case: each <MFBridgeSSR> has its own DOM node, so events stay scoped to it. No filtering by instance id, no manual subscription bookkeeping. Warming the Cache From RSC If you know a fragment is going to be needed, you can start the fetch before <MFBridgeSSR> even renders. Suspense then skips the fallback entirely: TypeScript-JSX import { preloadFragment } from '@mf-toolkit/mf-ssr' // In a Server Component or route loader preloadFragment('https://checkout.acme.com/fragment', { orderId }) By the time the component renders down the tree, the HTML is already there. Where It Fits If your microfrontends share one build (a single bundler config that imports every remote), you don't need any of this. Use whatever your framework gives you. mf-ssr is for the case where each team builds and deploys independently. Different repos or not, the point is that there is no shared build step pulling everything into one Node process — and you still want a full page on first paint. The bet is that HTTP is a good enough boundary between teams, and that DOM events are a good enough way to keep host state in sync with remote rendering after hydration. The CSS isolation question, by the way, lives in mf-bridge, not here: it has shadowDom and adoptHostStyles props that wrap the remote in a Shadow DOM and forward host stylesheets (including Tailwind / CSS-in-JS chunks injected after mount) into the shadow root. SSR fragments don't use it by default since the HTML is inlined into the host response, but the option exists if you want it. Try It The package is published as @mf-toolkit/mf-ssr. The repo has runnable examples, and I've also made a demo repo where you can play with all my tools. If you've solved the same problem in a different way, I'd be curious to compare notes.