Frameworks

Need to perform asynchronous operations and support multitasking in your app? Async/await is at your service — simple and elegant. The cooperative thread pool efficiently switches threads between tasks, while the compiler ensures thread safety at the type level. You can even seamlessly bridge older parts of your codebase written in GCD! But then, for some reason, your app starts hanging in production… Below, we will explore specific examples (complete with diagrams) of how not to mix async/await code with DispatchQueue (the same rules apply to other blocking primitives). The Root of the Problem The system doesn’t allocate a dedicated thread for every Task. Instead, tasks are executed on a cooperative thread pool, where the number of available threads never exceeds the number of active CPU cores. Therefore, you can cheaply spawn thousands of tasks — they are merely small allocations on the heap, not separate threads. However, a blocking GCD call or an infinite task (a loop) is not a suspension point; they occupy the thread and do not return it to the pool. The more of these tasks there are, the higher the chance of depleting the pool. Each of the methods below leads to this situation in its own way. Method #1. Saturating the Pool With Blocking Tasks The simplest way is to occupy every thread in the pool with a task that blocks it until its execution is complete. An example using DispatchQueue.sync: Swift // The pool size is 2. We launch 2 blocking tasks, each on its own queue. for i in 0..<2 { Task { DispatchQueue(label: "blocking-\(i)").sync { // blocks the thread // some heavy work } print("done") } } Task { print("See you later...") } // stuck in the queue, won't execute anytime soon The task inside sync does not suspend. The thread from the pool waits until the block finishes executing on the queue. If you do this simultaneously on every thread in the pool, its throughput will drop to zero. The pool recovers after the blocks complete. But while they are executing, nothing that lives on it makes progress: non-isolated async functions, regular actors, TaskGroup (@MainActor and GCD queues continue to work in the meantime — the main actor has its own executor on the main thread, and GCD has its own pool). The heavier the task — a synchronous network request, heavy computation, file I/O — the longer the stall. How can this slip through tests? If you only test on powerful devices. If, say, 4 blocking tasks occur simultaneously at runtime, the code might run normally on 8 cores, but then fail on a 2-core CI runner or a low-end device. Additionally Pool exhaustion due to blocking calls is discussed in the Swift Forums thread Deadlock When Using DispatchQueue from Swift Task, where a reader-writer subsystem managed by a TaskGroup deadlocks as soon as a sufficient number of tasks simultaneously block their threads in the pool. The Problem With the Vision Framework A blocking call can be inside third-party code, and you won’t see it in your own. The Swift Forums thread Cooperative pool deadlock when calling into an opaque subsystem describes such a case: a seemingly synchronous Apple API (VNImageRequestHandler.perform from Vision) internally drops down into GCD and blocks the calling thread. Just a few concurrent tasks calling it are enough to exhaust the cooperative pool and hang the entire application. Method #2. Creating a Deadlock Between Queues Thread starvation is temporary if the blocking call eventually finishes. To make it permanent, you need to arrange it so that two blocked threads wait for each other. Swift let queueA = DispatchQueue(label: "A") let queueB = DispatchQueue(label: "B") Task { queueA.sync { // holds the pool thread on A... queueB.sync { } // ...then waits for B } } Task { queueB.sync { // holds the pool thread on B... queueA.sync { } // ...then waits for A → circular wait } } The queueA block won't complete until queueB is freed, and the queueB block won't complete until queueA is freed. Important: This is not a guaranteed deadlock. It only happens if both outer sync calls manage to capture their queues before the inner sync calls execute. If the first task completely finishes before the second one starts, nothing will happen. This can lead to intermittent (flaky) bugs. Method #3. Creating a Deadlock on a Single Queue Variant 1. Two Nested sync Calls A familiar situation: Swift let queue = DispatchQueue(label: "serial") Task { queue.sync { // blocks the cooperative thread // ...work... queue.sync { } // sync on the same serial queue } } In practice, this will more likely result in a crash rather than a hang. libdispatch recognizes the simple case — the thread already owns the queue and calls sync on it again — and intentionally crashes the application with EXC_BAD_INSTRUCTION and the message BUG IN CLIENT OF LIBDISPATCH: dispatch_sync called on queue already owned by current thread. This applies to a serial queue. A nested sync on a concurrent queue will not cause a deadlock, but it will still hold the pool thread. sync deadlocks between queues and on a single queue are well-known GCD "pitfalls"; it is easy to fall into them in a cooperative pool of limited size. Variant 2. A Hidden Reentrant sync and a Single Queue for Everything The blocking call can be hidden behind an innocent helper function. For example, in a seemingly safe synchronous accessor like this: Swift let queue = DispatchQueue(label: "store") func currentUser() -> User { // used throughout the code queue.sync { _user } // fine — as long as you are not on `queue` } And now someone somewhere starts work on the same queue and calls this helper from within: Swift Task { queue.sync { // now executing ON `queue` let user = currentUser() // currentUser() calls queue.sync again apply(user) // the same serial queue → crash } } Each call looks normal on its own. The problem only arises when they are combined, and its two halves might reside at opposite ends of the codebase. As a result, the application crashes with the same libdispatch message as in Variant 1, but the stack trace doesn't immediately reveal that two "normal" halves of code from different files are to blame. Method #4. Not Keeping Track of @MainActor The main thread is not part of the cooperative pool; @MainActor has its own executor on the main thread. But the scheduling model is the same — cooperative — and a blocking sync breaks it in exactly the same way: Swift @MainActor func onTap() { let worker = DispatchQueue(label: "load") worker.sync { // blocks the main thread, the UI freezes let data = loadDataSync() DispatchQueue.main.sync { // worker is now waiting for main... render(data) // ...but main is blocked above → deadlock } } } Blocking the main thread stops rendering, gesture processing, and run loop events. The user sees a frozen screen, and the watchdog might kill the application. Method #5. Not Suspending Heavy Synchronous Tasks Without GCD or any primitives. A task performing long synchronous work between await points also does not yield its thread back: Swift Task { while true { heavySynchronousWork() // never reaches an await } // holds its thread forever } In a cooperative pool, the runtime can only reassign a thread at a suspension point. No await means no yielding. From the pool's perspective, a tight CPU loop without an await is indistinguishable from a blocking call; it just does useful work while 'starving' everyone else. A possible solution is to break the long-running work into chunks with await Task.yield() between them: Swift Task { while !Task.isCancelled { heavySynchronousWork() await Task.yield() } } Apple’s documentation for Task.yield() describes it as suspending the current task to allow other tasks to execute. But this is not an ideal solution, because between yield points, the work still occupies a pool thread. There is another option: moving the heavy work out of the pool entirely, for example, via GCD + continuation or a separate executor. How Not to Break Swift Concurrency Do not call long-running tasks under blocking primitives or queue.sync inside a Task. Short critical sections under a fast lock (os_unfair_lock, NSLock, an instantaneous queue.sync around a field read) are acceptable: the thread holding the lock will perform the work itself and release it immediately.Call callback APIs using continuations. To turn a GCD API with a completion handler into an async function, wrap it in withCheckedContinuation (or withCheckedThrowingContinuation when an error is possible). The continuation suspends the task and resumes it from the callback without blocking the thread.Keep blocking sync calls from the same queue in one place. If a public function blocks the thread, indicate this explicitly (via its signature or a comment) or use async.Watch out for calls within @MainActor methods. Do not call heavy tasks under sync on the main thread, with the exception of a short sync for the sake of an atomic read. Launch heavy work in a separate Task or queue and update the UI asynchronously.Use suspension in heavy loops. Insert await Task.yield() so that a long (or infinite) task does not hijack a pool thread for itself, or move the work out of the cooperative pool.Test on low-end devices and under load. In an environment with 1–2 cores or on a pool saturated with concurrent tasks.

My first attempt to deploy a Spring Boot microservice on AWS Fargate didn’t fail loudly. It failed quietly — in a loop. ECS kept launching tasks, the Application Load Balancer kept marking them unhealthy, and the service never stabilized. The logs looked fine, the container looked fine, but the ALB replaced every task within seconds. The root cause was painfully simple: Spring Boot needed 45 seconds to start, and my ALB health‑check timeout was 5 seconds. The tasks never had a chance. That night changed how I build and deploy microservices. It forced me to rethink startup behavior, JVM sizing, networking, task definitions, and the entire CI/CD pipeline. This article is the guide I wish I had before that incident — a practitioner’s walkthrough of deploying a production‑ready Spring Boot service on AWS Fargate, with real artifacts and the details that matter when things go wrong. The Architecture That Finally Worked Once the health‑check issue was fixed, the architecture settled into a predictable, cloud‑native flow: Developers push code to GitHubGitHub Actions builds the JARDocker image is built and pushed to Amazon ECRECS service runs AWS Fargate tasksTraffic enters through an Application Load BalancerTasks run in private subnetsConfiguration comes from Parameter Store and Secrets ManagerLogs and metrics flow to CloudWatch It’s the standard modern microservice pipeline — but the difference between “standard” and “production‑ready” is in the details. The Spring Boot Service The microservice itself was simple — a REST API with a few endpoints. The real complexity wasn’t the controller logic; it was everything around it: startup time, health checks, configuration management, and container behavior under load. A Dockerfile Built for Production My first Dockerfile looked like the one many tutorials start with: a single‑stage build running as root with no JVM tuning. It worked locally but failed under real load. Fargate tasks with default JVM heap sizing inside a 2GB container are a classic OOM story. Here’s the hardened version that finally stabilized deployments: Dockerfile FROM eclipse-temurin:21-jre # Create non-root user RUN useradd -u 1001 springuser WORKDIR /app # Layer extraction for faster builds COPY target/*.jar app.jar # JVM tuning for Fargate ENV JAVA_OPTS="\ -XX:MaxRAMPercentage=75 \ -XX:+UseContainerSupport \ -XX:+ExitOnOutOfMemoryError \ " USER springuser ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"] This eliminated the OOMKilled events I saw on 2GB tasks and made startup time predictable. Pushing to Amazon ECR With Real Commands The first time I wrote down my ECR commands, they were placeholders. In production, they need to be exact: C aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 docker push \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 Immutable semantic version tags make rollbacks predictable and prevent “latest‑tag roulette.” The ECS Task Definition That Actually Runs in Production A real Fargate deployment lives or dies by its task definition. Here’s the JSON I use today — including secrets pulled from Parameter Store and Secrets Manager: JSON { "family": "employee-service", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "512", "memory": "1024", "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/employeeServiceRole", "containerDefinitions": [ { "name": "employee-service", "image": "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3", "portMappings": [ { "containerPort": 8080, "protocol": "tcp" } ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:ssm:us-east-1:<ACCOUNT_ID>:parameter/db/password" }, { "name": "API_KEY", "valueFrom": "arn:aws:secretsmanager:us-east-1:<ACCOUNT_ID>:secret:thirdparty/api" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/employee-service", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ] } The ALB Health Check That Stopped the Outage My outage happened because the ALB was impatient. Here’s the configuration that finally stabilized deployments: settingvalue Path /actuator/health Interval 20 seconds Timeout 10 seconds Healthy threshold 3 Unhealthy threshold 3 Spring Boot startup time + ALB patience = stable deployments. Why Fargate Tasks Belong in Private Subnets Early on, I deployed tasks in public subnets because it felt simpler. It wasn’t. Public IPs meant the containers were directly reachable from the internet — port scans, bot traffic, and noisy logs. Moving tasks to private subnets solved several problems at once: Reduced Attack Surface No public IPs. No direct inbound traffic. Only the ALB can reach the tasks. A Single Secure Entry Point The ALB handles TLS termination, redirects HTTP→HTTPS, performs health checks, and integrates with WAF. Clients never bypass it. Cleaner Security Groups ALB SG: inbound 443 from the internetTask SG: inbound only from ALB SG Nothing else touches the containers. Compliance Alignment PCI, SOC 2, HIPAA — all prefer minimizing public exposure. Controlled Outbound Access Tasks use a NAT Gateway for outbound calls (updates, third‑party APIs) without exposing themselves. Better Scalability ALB target groups automatically track tasks across AZs as ECS scales. The architecture becomes simple and predictable: Internet → ALB (public subnets) → Fargate tasks (private subnets) It’s quieter, safer, and easier to operate. The GitHub Actions Workflow That Deploys Automatically Here’s the pipeline that builds, tests, pushes, and deploys the service: YAML name: Deploy to Fargate on: push: branches: ["main"] jobs: build-deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up JDK uses: actions/setup-java@v4 with: java-version: "21" - name: Build JAR run: mvn -B clean package - name: Login to ECR uses: aws-actions/amazon-ecr-login@v2 - name: Build and Push Image run: | docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 ${{ env.ECR_REGISTRY }/employee-service:1.0.3 docker push ${{ env.ECR_REGISTRY }/employee-service:1.0.3 - name: Deploy ECS Service uses: aws-actions/amazon-ecs-deploy-task-definition@v2 with: task-definition: ecs-task.json service: employee-service cluster: prod-cluster Auto Scaling With Real Target Tracking JSON Target tracking is the simplest and most reliable scaling strategy for Fargate: JSON { "TargetValue": 50.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilization" }, "ScaleOutCooldown": 30, "ScaleInCooldown": 60 } I use 50% as the target because it balances cost and responsiveness. What I Learned Every failure taught me something: ALB timeouts taught me to respect startup timeOOMKilled tasks taught me to tune the JVMPublic subnets taught me to isolate workloadsManual deployments taught me to automate everything AWS Fargate really does deliver on its promise — no servers to manage, automatic scaling, and clean integration with ECS — but only after you learn the hard parts. If you’re deploying Spring Boot on Fargate, I hope you learn those lessons from this article instead of from your own outage.

Picture the scene: One of the services in your backend is a mature Django app that no one has the resources, time, or, frankly, the will to rewrite. The ORM, the admin panel, and the broader ecosystem all earn their keep. But you’re looking for the best way to describe your API, and FastAPI catches your eye. It looks like a great fit: native typing, pydantic-based validation, OpenAPI out of the box, and of course the support for async endpoints. That's the situation our team found itself in - we decided to use both frameworks and take from each what suited us best. Not everything went smoothly — this post is what we built, what broke afterward, and what we learned. The First Win So we wired it up, and it works. FastAPI runs as the ASGI application, and the existing Django app plugs into it. Python # asgi.py import os os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings") from django.core.asgi import get_asgi_application from fastapi import FastAPI app = FastAPI() django_app = get_asgi_application() app.mount("/legacy", django_app) Great! Now: Both Django and FastAPI endpoints live side by side, with no pressure to refactor everything in a single day — that was important for us.In the new parts of the app, Django steps back into a single role: communicating with the database through its models.Endpoints can be either sync or async. That was the win. But there was the other side also. Pitfall 1: Async Endpoints Started Running One at a Time When you reach out to external services, chances are you also want to enrich the request with something from your database, or save the result back to it (we did). Here's a tiny example: A single async handler that fetches data about Order from the database (we use Postgres) and forwards it to an external payment provider. Python from asgiref.sync import sync_to_async from fastapi import FastAPI app = FastAPI() @app.post("/orders/{order_id}/dispatch") async def dispatch_order(order_id: int) -> OrderDTO: order = await sync_to_async(get_order)(order_id) # fetch from DB await client.send_order(order) # call external service return order # code that uses a Django model def get_order(order_id: int) -> OrderDTO: order = Order.objects.get(id=order_id) return OrderDTO(id=order.id, amount=order.amount) Inside an async function, you can’t call the Django ORM synchronously. The documented approach is sync_to_async, which moves the synchronous call to a separate thread so it doesn’t block the event loop. Now let's see what happens under concurrent load. Drop a three-second sleep into get_order: Python from django.db import connection def get_order(order_id: int) -> OrderDTO: order = Order.objects.get(id=order_id) with connection.cursor() as cursor: cursor.execute("SELECT pg_sleep(3);") return OrderDTO(id=order.id, amount=order.amount) And fire three requests in parallel: Shell URL="http://localhost:8000/orders/1/dispatch" curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" & curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" & curl -s -o /dev/null -X POST -w "%{time_total}s\n" "$URL" >> 3.012s >> 6.024s >> 9.037s We expected ~3 seconds and got nine. The handlers ran one after another, not concurrently. And if you log the thread and database connection IDs from inside get_order, all three requests print the same values. Why? By default sync_to_async(get_order) runs with thread_sensitive=True, which means the function runs in the same thread as all other thread_sensitive functions. A standalone Django ASGI app does extra work here: it opens a fresh context per request, so requests run in parallel. The benchmark suggests that in our setup FastAPI doesn't: all three sync_to_async calls land on the same thread and line up one behind another. The event loop itself stays free, by the way: a purely async route keeps responding while the three /dispatch requests wait in that queue. But three async handlers with ORM calls queue up on the same thread, sharing the same connection. For a moment we hoped Order.objects.aget(...) or other Django async ORM helpers would save us here. They won't: for now under the hood they call the same sync_to_async. Can we just flip to sync_to_async(..., thread_sensitive=False)? Probably not - it is not a safe default. Django carries a lot of per-request state in thread-locals: the current DB connection, transaction.atomic(), etc. The Django docs say: "a lot of existing Django code assumes it all runs in the same thread." What to Do About It No silver bullet, but two approaches hold up: Split handlers by what they touch. Reserve async def for endpoints that genuinely don't touch the ORM — async-native HTTP calls, cache reads, etc. For ORM-bound endpoints, declare them as plain sync routes. FastAPI runs sync routes on its thread pool, so they actually run in parallel, and each thread gets its own Django connection. As long as these endpoints don't make many slow external calls, this can work.Move the work out of the handler entirely. If your project already runs with a message broker, the possible answer to "external API + DB write inside a handler" is to stop doing it inside a handler at all. Drop an event on the bus, let consumers handle the side effects, return immediately. The catch: this only makes sense when an event-driven flow already fits your system — because it is, of course, no small refactor. Pitfall 2: Tests That Can't See Their Own Data Now let's write a test for get_order — a sync endpoint that reads an order from the DB. The test runs with pytest-django: we create an order in the database and call the handler. Python # app.py import pytest from fastapi import FastAPI, HTTPException from fastapi.testclient import TestClient app = FastAPI() @app.get("/orders/{order_id}") def get_order(order_id: int) -> OrderDTO: try: order = Order.objects.get(id=order_id) except Order.DoesNotExist: raise HTTPException(status_code=404) return OrderDTO(id=order.id, amount=order.amount) @pytest.mark.django_db def test_get_order(): Order.objects.create(id=1) response = TestClient(app).get("/orders/1") assert response.status_code == 200 # and we'll have 404 You get 404 Not Found. The handler ran, looked at the database, and the order was nowhere to be found. Four facts conspire here: Pytest runs your test's data setup in one thread; when the FastAPI test client calls the endpoint, the handler runs in another.pytest-django wraps every test in an open transaction and rolls it back at the end. That's how the suite stays fast and isolated. The transaction lives on a single database connection.Django opens a database connection per thread.Postgres defaults to READ COMMITTED isolation: one connection cannot see another connection's uncommitted writes. So: the test body runs in the pytest thread. Its Order.objects.create(...) uses connection 1, inside pytest-django's open transaction. When TestClient hits the endpoint, FastAPI dispatches the handler to a worker thread from its thread pool, on another thread with its own connection 2. Connection 2 looks at the database and sees no order, because connection 1 hasn't committed, so connection 1's write is effectively invisible to everyone else. Again — What to Do? Test in layers. Unit-test the endpoint contract with the ORM mocked - those tests don't cross thread or connection boundaries, so the visibility problem simply can't appear. Test business logic and data access in their own tests, without going through TestClient. For cases when the full end-to-end test is still needed - the commonly suggested fix is @pytest.mark.django_db(transaction=True). This switches the test to a mode where writes actually commit, so other connections can see them. But it has its cost: pytest-django now does a database flush after every test, and the suite gets noticeably slower. On a large suite, for us "noticeably" meant minutes - too much on every run, so we use it only for exceptional cases. The Recap FastAPI brings obvious wins — OpenAPI docs, clean endpoint code, typing all the way through; Django gives you a greatly tested ORM and admin. Putting them in the same process gives us both — and a thread-and-connection model that doesn't behave the way we'd expect. Budget for the architecture work before you budget for the migration. Was it worth it? Yes — we got the clean, typed API we were after, and we kept Django's ORM instead of porting the whole data layer to another framework. Would we do it again? Not sure. The trade-offs of this integration may outweigh its benefits for us, so other combinations might be a better fit. If you’ve run into the same solution and found an approach with better trade-offs, please share; the comments are open. Reproduce it yourself. An example with a benchmark and failing tests is in https://github.com/evchibisova/fastapi-over-django-test.

The Problem With "Just Add More Workers" Most Spark performance issues on Databricks aren't solved by scaling the cluster — they're caused by shuffle and skew, and no amount of extra nodes fixes a badly partitioned join. This post builds a realistic pipeline (order events joined against a small dimension table, aggregated, and written to Delta Lake) from the ground up, and uses it to work through: How Spark's shuffle actually behaves during a wide transformationDiagnosing and fixing data skew with salting and adaptive query execution (AQE)Laying out the resulting Delta table with Z-Ordering so downstream queries skip irrelevant filesGoverning access to the whole pipeline with Unity Catalog Architecture Overview Pipeline shape – a batch job reading raw events, joining against a dimension table, aggregating, and writing to a governed Delta table: What happens inside a shuffle stage – this is the part most tutorials skip, and it's the key to understanding why skew hurts: Step 1: Set Up Governed Tables in Unity Catalog Everything downstream depends on tables being registered under Unity Catalog, which gives you centralized access control and lineage instead of per-workspace table grants. SQL -- setup.sql, run in a Databricks SQL or notebook cell CREATE CATALOG IF NOT EXISTS retail_analytics; CREATE SCHEMA IF NOT EXISTS retail_analytics.events; CREATE TABLE IF NOT EXISTS retail_analytics.events.raw_orders ( order_id STRING, customer_id STRING, product_id STRING, quantity INT, event_ts TIMESTAMP ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/raw_orders'; CREATE TABLE IF NOT EXISTS retail_analytics.events.dim_products ( product_id STRING, category STRING, unit_cost DOUBLE ) USING DELTA LOCATION 'abfss://data@<storage-account>.dfs.core.windows.net/dim_products'; GRANT SELECT ON TABLE retail_analytics.events.raw_orders TO `analysts`; Step 2: Read and Force a Broadcast Join for the Small Dimension Table dim_products is small, so let Spark broadcast it rather than shuffle both sides of the join. Python # pipeline.py from pyspark.sql import functions as F orders = spark.table("retail_analytics.events.raw_orders") products = spark.table("retail_analytics.events.dim_products") Without the explicit broadcast hint, Spark's cost-based optimizer usually picks a broadcast join automatically for small tables, but being explicit avoids surprises when the dimension table grows past spark.sql.autoBroadcastJoinThreshold (default 10MB) without anyone noticing. Step 3: The Aggregation That Triggers a Shuffle groupBy on customer_id is a wide transformation — Spark must shuffle rows so all records for a given key land on the same reducer. Python agg = ( joined .groupBy("customer_id", "category") .agg( F.sum(F.col("quantity") * F.col("unit_cost")).alias("total_spend"), F.count("order_id").alias("order_count"), ) ) If one customer_id (say, a test account or a large B2B buyer) accounts for a disproportionate share of rows, this is where skew shows up: one reducer task runs for minutes while the rest of the stage finishes in seconds. You'll see this in the Spark UI as a single long-running task in an otherwise short stage. Step 4: Fixing Skew With Salting Adaptive Query Execution (AQE) handles a lot of skew automatically in modern Databricks Runtime, but for known hot keys, explicit salting is still the most predictable fix. Python from pyspark.sql import functions as F import random SALT_BUCKETS = 20 # Add a salt column to spread the hot key across multiple reducers salted = joined.withColumn("salt", (F.rand() * SALT_BUCKETS).cast("int")) partial_agg = ( salted .groupBy("customer_id", "category", "salt") .agg( F.sum(F.col("quantity") * F.col("unit_cost")).alias("total_spend"), F.count("order_id").alias("order_count"), ) ) # Second pass: combine the salted partial aggregates into the final result final_agg = ( partial_agg .groupBy("customer_id", "category") .agg( F.sum("total_spend").alias("total_spend"), F.sum("order_count").alias("order_count"), ) ) This two-phase pattern — pre-aggregate on a salted key, then combine — is the same trick used inside combiners in older MapReduce systems. It trades a bit of extra shuffle for eliminating the single-reducer bottleneck. Also worth setting explicitly rather than relying on the default of 200: Python spark.conf.set("spark.sql.shuffle.partitions", "auto") # let AQE size it dynamically spark.conf.set("spark.sql.adaptive.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") Step 5: Write to Delta Lake With Optimized Writes Python ( final_agg.write .format("delta") .mode("overwrite") .option("delta.autoOptimize.optimizeWrite", "true") .saveAsTable("retail_analytics.events.customer_spend_summary") ) Optimized writes shuffle data before writing so you get fewer, better-sized files instead of many small ones — this costs some write-time latency in exchange for faster reads later. Step 6: Z-Order the Table for the Queries That Matter If most downstream queries filter on customer_id, co-locate related rows physically so Spark can skip files that can't match the filter. Python OPTIMIZE retail_analytics.events.customer_spend_summary ZORDER BY (customer_id); Z-ordering aims to produce evenly balanced data files by row count rather than raw size, and its effectiveness depends on the column having reasonably high cardinality — Z-ordering by a low-cardinality column like category alone gives little benefit. On newer Delta Lake / Databricks Runtime versions, Liquid Clustering is generally the preferred choice for new tables since it adapts as query patterns change, while ZORDER remains relevant mainly for existing tables not yet migrated. SQL -- Preferred on new tables (Databricks Runtime supporting Liquid Clustering): CREATE TABLE retail_analytics.events.customer_spend_summary CLUSTER BY (customer_id); Comparing Shuffle-Mitigation Techniques TechniqueFixesCostWhen to useBroadcast joinShuffle on the large side of a joinExtra memory on executorsSmall (<~10MB by default) dimension/lookup tablesAQE skew join handlingAutomatic detection of skewed partitionsMinor planning overheadDefault-on; good general safety netManual saltingKnown, severe hot keysExtra shuffle for the two-phase aggregateHot keys that AQE doesn't fully resolveRepartition by keyUneven task distribution before a shuffle stageOne extra shufflePre-shaping data before multiple downstream joinsZ-OrderingSlow reads due to unnecessary file scansShuffle + rewrite during OPTIMIZEExisting tables, high-cardinality filter columnsLiquid ClusteringSame as Z-Order, plus evolving query patternsOngoing background clusteringNew tables on supporting runtime versions Governance: Tying It Back to Unity Catalog Because every table above was created under retail_analytics, access control, audit logging, and lineage are handled centrally rather than per-cluster: SQL -- Restrict PII-adjacent columns without duplicating the table CREATE VIEW retail_analytics.events.customer_spend_summary_masked AS SELECT customer_id, category, total_spend, order_count FROM retail_analytics.events.customer_spend_summary; GRANT SELECT ON VIEW retail_analytics.events.customer_spend_summary_masked TO `bi_readers`; Production Considerations Diagnose before tuning. Check the Spark UI's stage view for one long-running task among many short ones — that's the signature of skew, not just "the job is slow."Predictive optimization can run OPTIMIZE and ANALYZE automatically on Unity Catalog-managed tables, which reduces the need for scheduled maintenance jobs for many workloads.Don't Z-Order everything. It requires shuffling and rewriting the table (or partition), so reserve it for columns that are actually filtered on frequently downstream. References Best practices: Delta Lake — Azure Databricks / Microsoft LearnData skipping (Z-ordering) — Azure Databricks / Microsoft LearnOptimizations — Delta Lake documentationDelta Lake Under the Hood: What Every Data Engineer Should Know — Databricks CommunityMastering Delta Lake Performance: Z-Ordering vs Liquid Clustering — Medium

By Jubin Soni, FBCS CORE

Language models become much more useful when they can answer questions about information they were never trained on, including your internal documentation, product manuals, policies, and other proprietary data. Prompting alone cannot solve this, because the model simply does not have access to that knowledge. Retrieval-Augmented Generation, or RAG, is the most common way to bridge that gap. Spring AI comes with solid support for building RAG systems. It has been almost three years since Spring AI showed up, and in that time it has grown from an experimental member of the Spring portfolio into a mature layer over chat models, embedding models, vector stores, and the plumbing that sits between them, which happen to be exactly the pieces a RAG system needs. In this article, we build a small but complete RAG service with Spring AI 2.0. The application reads a set of documents into a PostgreSQL vector store, retrieves the fragments that are relevant to a user question, and lets Anthropic's Claude put together the answer based on those fragments. Everything runs from a standard Spring Boot project, and every step can be reproduced on macOS, Windows, or Linux. The full project is available on GitHub. If you just want to see the finished result, or you would rather skip the step-by-step build below, you can clone the repository and run it as it is. Everyone else can follow along and generate this project from scratch. The prompts themselves are kept deliberately simple. You can tune retrieval and prompts forever; here we care about the architecture and how the pieces fit together in Spring. Approach RAG is not really a single feature. It is more of a small pipeline, and the code below makes a lot more sense once its parts have names. Embedding: a vector of numbers that captures the meaning of a piece of text. Texts that mean similar things end up with vectors that are close to each other.Embedding model: the model that computes these embeddings. It is a different model from the chat model, and it has a different job.Vector store: a database that keeps text fragments together with their embeddings and can answer the question, "which stored fragments are closest in meaning to this query?"Chunking: documents are too large to embed and retrieve as a whole, so we split them into smaller fragments (chunks) before storing them.Similarity search: we embed the user question and fetch the top-k closest chunks from the store.Augmentation: we append the retrieved chunks to the user question before sending it to the chat model, so the model answers from the context we provided instead of from its training data. One thing here is worth calling out, because it shapes the whole setup of the project: the LLM model used in chat and the embedding model are two separate choices. As of today, Anthropic offers LLM models but no embedding API, so a Claude-based RAG system always has to pair Claude with an embedding model from somewhere else. Rather than bringing in a second cloud provider and a second API key, this project computes embeddings locally (inside the JVM), using Spring AI's ONNX transformers module and the well-known all-MiniLM-L6-v2 sentence transformer. It is free and fast enough for this, and it keeps everything on one API key. In our scenario, the service is an internal assistant for a fictional company called Nimbusfield Systems, and it answers employee questions based on the company handbook. The company and the handbook are fictional on purpose. Claude cannot possibly know about it, which makes it easy to verify that the answers really come from our documents and not from the model's own memory. We build this in three steps: Expose a /ask endpoint backed by Claude, with no retrieval, and show that the model cannot answer handbook questions.Ingest the handbook into PGvector at application startup: read, chunk, embed, and store.Attach Spring AI's QuestionAnswerAdvisor to the same ChatClient and ask again. Prerequisites Java 21Maven 3.9.x (the Maven wrapper included in generated projects works too)Spring Boot 4.0.xSpring AI 2.0.0Docker Desktop (macOS/Windows) or Docker Engine (Linux), used only to run PostgreSQL. A project skeleton can be generated at start.spring.io by selecting Web, Anthropic Claude, PGvector Vector Store, and Docker Compose Support. The remaining Spring AI modules are added manually below. The Claude API Key Sign in (or sign up) at the Anthropic Console, open Settings, then API Keys, and create a new key. New accounts may need a small prepaid credit before the API accepts requests, but the runs in this article cost only a few cents. The key is shown only once, so store it right away as an environment variable. If you would rather not spend anything at all, you can still follow along and read through the steps without running the calls yourself. macOS/Linux: export ANTHROPIC_API_KEY=sk-ant-... Windows (PowerShell, persists across sessions after reopening the terminal): setx ANTHROPIC_API_KEY "sk-ant-..." Solution Dependencies With the Spring AI BOM in place, there is no need to repeat versions on the individual artifacts. Initializr expresses the BOM's own version as a property rather than a hardcoded literal, so there is a single place to bump it later: XML <properties> <java.version>21</java.version> <spring-ai.version>2.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> A common source of confusion is that start.spring.io has no dependency literally named "Spring AI." Each provider- or store-specific starter (Anthropic Claude, PGvector Vector Database, and so on) is itself a Spring AI module, and picking one transitively pulls in the framework's core classes. (like ChatClient, VectorStore, etc.) Selecting any one of them is also what makes Initializr add the spring-ai-bom as shown above to the generated pom.xml for you. The BOM itself is never a separate item you tick on the Initializr dependency screen. The application needs six Spring AI modules on top of the web starter, each one with a single responsibility. XML <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> <!-- Chat model: Anthropic Claude --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-anthropic</artifactId> </dependency> <!-- Embedding model: local ONNX sentence transformer --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-transformers</artifactId> </dependency> <!-- Vector store: PostgreSQL + pgvector --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-vector-store-pgvector</artifactId> </dependency> <!-- RAG advisor --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-vector-store-advisor</artifactId> </dependency> <!-- Document reading (PDF, Word, Markdown, HTML, and more) --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-tika-document-reader</artifactId> </dependency> <!-- Starts the database container on application startup --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <!-- Docker Compose service connections for Spring AI vector stores --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> </dependencies> Two models are referenced from the code here. One is the chat model, Claude, which is served from the Anthropic API. The other is the embedding model, which runs locally, right inside the application. We will look at that local embedding model in the next section. The Embedding Model By default, the transformers starter fetches tokenizer.json and model.onnx from Spring AI's own GitHub repository the first time the application starts and then caches them locally. In practice, this default setup is a bit fragile. raw.githubusercontent.com may rate-limit unauthenticated requests, and model.onnx (which is roughly 90 MB) is stored via Git LFS, whose bandwidth quota can run out independently of the ordinary rate limit. When that happens, the endpoint serves the small LFS pointer stub instead of the binary, with a normal-looking HTTP 200, and the failure only shows up later as a cryptic ONNX Runtime protobuf-parsing error rather than a clear download error. The fix is to bundle both files with the application instead of fetching them at startup. So we download them once: Shell mkdir -p src/main/resources/onnx/all-MiniLM-L6-v2 curl -fL -o src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json \ https://raw.githubusercontent.com/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json curl -fL --http1.1 -o src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx \ https://media.githubusercontent.com/media/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx Then we point the embedding model at these local files in our application.properties, overriding the GitHub-backed defaults: Properties files spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json With these two properties set, the application never touches the network for the embedding model, neither on the first run nor on any run after it. The Database The pgvector team publishes a PostgreSQL image with the extension already installed. A compose.yaml in the project root is all we need: YAML services: pgvector: image: "pgvector/pgvector:pg17" environment: - "POSTGRES_DB=nimbusfield" - "POSTGRES_USER=nimbusfield" - "POSTGRES_PASSWORD=nimbusfield" labels: - "org.springframework.boot.service-connection=postgres" ports: - "5432" The labels entry is important. Spring Boot's Docker Compose support auto-detects connection details by matching the image name against a list of well-known images. Plain Postgres is on that list, but pgvector is not, since it is a third-party image. The label tells Spring Boot to treat this container as if it were the official Postgres image, and that is what actually makes the automatic connection wiring work. If we omit it, the container still starts, but Spring Boot never creates a ConnectionDetails bean for it, so the run fails with a connection error rather than falling back gracefully. Because spring-boot-docker-compose is on the classpath, running the application starts the container automatically and injects the connection details. This works the same way on macOS and Windows, as long as Docker Desktop is running. Anyone who prefers to manage the container manually can run the same image with docker run -p 5432:5432 .. and set the datasource properties explicitly. Configuration The complete application.properties, now including the embedding model overrides shown earlier: Properties files spring.ai.anthropic.api-key=${ANTHROPIC_API_KEY} spring.ai.anthropic.chat.model=claude-sonnet-5 spring.ai.anthropic.chat.max-tokens=1024 spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json spring.ai.vectorstore.pgvector.initialize-schema=true spring.ai.vectorstore.pgvector.dimensions=384 spring.ai.vectorstore.pgvector.index-type=HNSW spring.ai.vectorstore.pgvector.distance-type=COSINE_DISTANCE logging.level.org.springframework.ai.chat.client.advisor=DEBUG Four details matter here. First, max-tokens is mandatory for the Anthropic API, which caps every response explicitly. Spring AI provides a default, but it is better stated than left implied. Second, the two spring.ai.embedding.transformer.* properties point the embedding model at the local files we bundled in the previous section, instead of Spring AI's own GitHub-backed defaults. See "The Embedding Model" above for why this matters. Third, initialize-schema=true enables the automatic creation of the vector-store table and the required extensions. (Since Spring AI 1.0, this no longer happens silently by default.) Fourth, dimensions=384 must match the embedding model. all-MiniLM-L6-v2 produces 384-dimensional vectors. If the embedding model changes later, the table has to be recreated, because the column type is vector(384). The Documents Two short Markdown files under src/main/resources/docs play the role of the company handbook. remote-work-policy.md Markdown # Nimbusfield Systems Remote Work Policy Employees may work remotely up to three days per week. Remote days must be registered in the portal by Thursday of the preceding week. Working from abroad is permitted for a maximum of 30 calendar days per year and requires prior approval from both the line manager and the People team. travel-expenses.md: Markdown # Nimbusfield Systems Travel and Expenses The daily meal allowance for business trips is 65 EUR in Europe and 80 USD elsewhere. Taxi rides are reimbursed only between airports, hotels, and client sites. Flights longer than six hours may be booked in premium economy. All expense reports are due within 15 working days after the trip via the portal. Thanks to the Tika reader used below, dropping PDFs or Word documents into the same folder works without any code changes. Step 1: Chat Without Retrieval We start with a service that wraps a ChatClient, built once from the auto-configured builder: Java @Service public class AssistantService { private final ChatClient chatClient; public AssistantService(ChatClient.Builder builder) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .build(); } public String ask(String question) { return chatClient.prompt() .user(question) .call() .content(); } } And a controller associated with it: Java @RestController public class AssistantController { private final AssistantService assistantService; public AssistantController(AssistantService assistantService) { this.assistantService = assistantService; } @GetMapping("/ask") public ResponseEntity<String> ask(@RequestParam("question") String question) { return ResponseEntity.ok(assistantService.ask(question)); } } Start the application (./mvnw spring-boot:run on macOS/Linux, mvnw.cmd spring-boot:run on Windows) and ask it a handbook question: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The response, as we might expect, is: I don't have that information in my available knowledge base. Nimbusfield Systems' specific travel and expense policy—including per diem rates for European business trips—isn't something I can confirm accurately. To get the correct figure, please check: The company's Travel & Expense Policy document (likely on the intranet/HR portal)Your Finance or HR department directlyYour manager, if travel budgets are pre-approved per trip Would you like help with anything else I can assist with more reliably? This gives us a baseline. The model behaves correctly given what it knows, which is nothing at all about this company. Step 2: The Ingestion Pipeline Ingestion follows Spring AI's extract, transform, load structure: a DocumentReader extracts the text, a TextSplitter chunks it, and the VectorStore embeds and stores the chunks. The embedding call happens implicitly inside vectorStore.add() call. The auto-configured TransformersEmbeddingModel is wired into the PgVectorStore and each chunk is embedded into the table. Java @Component public class HandbookIngestion implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(HandbookIngestion.class); private final VectorStore vectorStore; private final JdbcTemplate jdbcTemplate; private final Resource[] handbook; public HandbookIngestion(VectorStore vectorStore, JdbcTemplate jdbcTemplate, @Value("classpath:docs/*.md") Resource[] handbook) { this.vectorStore = vectorStore; this.jdbcTemplate = jdbcTemplate; this.handbook = handbook; } @Override public void run(ApplicationArguments args) { Integer count = jdbcTemplate.queryForObject( "select count(*) from vector_store", Integer.class); if (count != null && count > 0) { log.info("Vector store already contains {} chunks, skipping ingestion", count); return; } TokenTextSplitter splitter = TokenTextSplitter.builder() .withChunkSize(300) .build(); for (Resource resource : handbook) { List<Document> documents = new TikaDocumentReader(resource).get(); documents.forEach(doc -> doc.getMetadata().put("source", resource.getFilename())); List<Document> chunks = splitter.apply(documents); vectorStore.add(chunks); log.info("Ingested {} chunks from {}", chunks.size(), resource.getFilename()); } } } The count check makes ingestion idempotent, so restarting the application does not duplicate every chunk. And the source metadata attached to each chunk enables filtered searches later, for instance restricting retrieval to a single document. That same idempotency check has a practical downside worth pointing out. Once the vector store has data, restarting the application will not pick up edits to the handbook files, since the count check short-circuits before the splitter ever runs. To force a clean re-ingestion, for instance after changing a handbook document, tear down the container together with its data volume, not just the container: docker compose down -v The chunk size of 300 tokens is generous for documents this small. The splitter's default of 800 is aimed at larger, real-world content. Chunking is the least exciting and yet the most important knob in a RAG system: chunks that are too large dilute the similarity signals, while chunks that are too small lose their context. It is worth experimenting here: try a few different chunk sizes and see how the system behaves. Just remember to run docker compose down -v between runs, so the vector store is rebuilt from scratch each time. Step 3: Attaching the Retrieval Advisor Now we come back to the plain AssistantService from Step 1 and upgrade it, rather than writing something new. The ChatClient wiring we built earlier stays and what changes is what gets attached to it. Spring AI models the cross-cutting concerns around a chat call as "advisors", which are conceptually close to interceptors. The QuestionAnswerAdvisor embeds the incoming user question, runs a similarity search against the vector store, and appends the retrieved chunks to the prompt before it reaches Claude. Enabling RAG is therefore a change to how the ChatClient is constructed, not to how the request is handled: Java public AssistantService(ChatClient.Builder builder, VectorStore vectorStore) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .defaultAdvisors( QuestionAnswerAdvisor.builder(vectorStore) .searchRequest(SearchRequest.builder() .topK(4) .similarityThreshold(0.5) .build()) .build(), new SimpleLoggerAdvisor()) .build(); } topK(4) retrieves at most four chunks per question, and similarityThreshold(0.5) discards weak matches, so an entirely unrelated question augments the prompt with nothing rather than with noise. The SimpleLoggerAdvisor, combined with the DEBUG logging property we set earlier, prints the fully augmented prompt. This is the single most useful debugging tool while tuning retrieval, because it shows exactly what Claude was given. We restart and repeat the same request: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The daily meal allowance for business trips in Europe is 65 EUR. Same model, same question, and this time a precise answer grounded in the retrieved handbook chunk instead of a generic deflection. The debug log confirms what is going on behind the scenes: the user question arrives at Claude wrapped in a prompt that contains the retrieved handbook fragments as context. Going Further The default behavior of QuestionAnswerAdvisor is usable, but there are two refinements worth implementing if you want to take this pattern further. The first one concerns grounding. Even with retrieved context, the model may fall back on its general knowledge when the context does not actually contain the answer. The advisor accepts a custom PromptTemplate that controls how the question and the context are merged, and this is the place to enforce stricter behavior. The template must contain the query and question_answer_context placeholders: Java PromptTemplate strictTemplate = PromptTemplate.builder() .template(""" {query} Answer strictly based on the context below. If the context does not contain the answer, reply exactly: "This is not covered by the handbook." --------------------- {question_answer_context} --------------------- """) .build(); QuestionAnswerAdvisor advisor = QuestionAnswerAdvisor.builder(vectorStore) .promptTemplate(strictTemplate) .build(); Asking about, say, the parental leave policy (which is absent from our two files) now produces the fixed refusal instead of an invention. If people are going to rely on it, you want this on. The second refinement could be structured output, and it composes cleanly with retrieval. Declaring a record and calling .entity() instead of .content() gives back a typed object, with Spring AI instructing the model to respond in the matching JSON schema: Java public record HandbookAnswer(String answer, String sourceHint, boolean coveredByHandbook) { } public HandbookAnswer askStructured(String question) { return chatClient.prompt() .user(question) .call() .entity(HandbookAnswer.class); } A last note on the embedding choice. A local MiniLM model is not the strongest embedding model available, and for a large multilingual corpus a hosted embedding API or a bigger ONNX model would retrieve better. This choice is easy to reverse: EmbeddingModel is an interface, swapping the implementation is a matter of a dependency and a property, and the only hard constraint is the one mentioned earlier: the vector dimensions in PGvector have to match whatever the embedding model produces. Conclusion In this article, we built the RAG flow step by step. We started with a plain chat endpoint that could not answer anything about the Nimbusfield handbook, because Claude had never seen it. We then ingested that handbook into PGvector, embedding each chunk locally, and attached Spring AI's QuestionAnswerAdvisor to the same client. That single change was enough to turn a generic model into a service that answers from your own documents. After that, we talked about how we can tighten the grounding, so the model says it does not know when the context has no answer, and pulled the response straight into a typed Java record. If you want to take it further, clone the project, point it at your own documents, apply further the techniques we discussed in the Going Further section, play with different chunk sizes, retrieval settings, and prompts to see how the answers change. The Spring AI documentation goes deeper into advisors, vector stores, and retrieval configuration. The complete, runnable project is available on GitHub.

By Murat Balkan CORE

If you've spent more than a year building enterprise Java apps, you've probably felt this specific kind of pain: a product manager asks for a new search filter, and you open your repository file to find it already has 18 methods. You write number 19, then 20, and somewhere around method 25 you start wondering if there's a better way. There is. It's called Spring Data JPA Specifications, and it's been sitting quietly in the framework the whole time. The Problem With Hard-Coded Query Methods Spring Data JPA's derived query methods are great for simple lookups. findByEmail is clean, readable, and requires zero SQL. But enterprise search rarely stays simple. Your CRM users want to filter customers by name and status. Then by date range. Then by city. Then by a keyword that could match name or email. Before long, you're maintaining a repository that looks like this: Java findByNameAndStatus(...) findByNameAndStatusAndCreatedDateBetween(...) findByNameOrEmailAndStatus(...) findByNameContainingIgnoreCaseAndStatusAndCreatedDateBetween(...) Each new requirement means a new method. The repository becomes a dumping ground. Testing it becomes a chore. Onboarding someone new becomes a conversation about which of the 30 methods to use. Specifications solve this by letting you define small, composable query predicates and combine them at runtime based on what filters the user actually provided. What a Specification Actually Is Under the hood, a Specification wraps the JPA Criteria API, the programmatic, type-safe way to build queries without writing raw SQL or JPQL. The Criteria API is powerful but verbose and tricky to read. Specifications give you that power with a cleaner surface area. Each Specification is just a lambda that produces a predicate: Java (root, query, criteriaBuilder) -> criteriaBuilder.equal(root.get("status"), "ACTIVE") That's it. One condition, one method, composable with anything else. Building It: A Customer Search Example Let's make this concrete. Imagine a Customer entity with name, email, status, and createdDate. Users can filter by any combination of these or none at all. The Entity Java @Entity public class Customer { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; private String name; private String email; private String status; private LocalDate createdDate; } A Specifications Utility Class Rather than scattering predicates across services, I keep them in a dedicated class: Java public class CustomerSpecifications { public static Specification<Customer> nameContains(String name) { return (root, query, cb) -> name == null ? null : cb.like(cb.lower(root.get("name")), "%" + name.toLowerCase() + "%"); } public static Specification<Customer> emailContains(String email) { return (root, query, cb) -> email == null ? null : cb.like(cb.lower(root.get("email")), "%" + email.toLowerCase() + "%"); } public static Specification<Customer> statusEquals(String status) { return (root, query, cb) -> status == null ? null : cb.equal(root.get("status"), status); } public static Specification<Customer> createdBetween(LocalDate start, LocalDate end) { return (root, query, cb) -> { if (start == null || end == null) return null; return cb.between(root.get("createdDate"), start, end); }; } } The null returns are intentional; Spring Data JPA ignores null predicates, which means you get automatic "skip this filter if not provided" behavior for free. The Repository Your repository needs to extend JpaSpecificationExecutor: Java public interface CustomerRepository extends JpaRepository<Customer, Long>, JpaSpecificationExecutor<Customer> { } Wiring It Together in the Service Java public List<Customer> searchCustomers(CustomerSearchRequest request) { Specification<Customer> spec = Specification .where(CustomerSpecifications.nameContains(request.getName())) .and(CustomerSpecifications.emailContains(request.getEmail())) .and(CustomerSpecifications.statusEquals(request.getStatus())) .and(CustomerSpecifications.createdBetween(request.getStartDate(), request.getEndDate())); return customerRepository.findAll(spec); } That single findAll call dynamically adapts to whatever combination of filters the caller provides. No branching logic, no 20 repository methods. When product asks for a fifth filter next sprint, you add one method to CustomerSpecifications and one .and() line in the service. Done. Going Further: OR Conditions, Joins, and Pagination OR Conditions The .or() combinator works exactly as you'd expect. A global search bar that checks name or email: Java Specification<Customer> spec = Specification .where(CustomerSpecifications.nameContains(keyword)) .or(CustomerSpecifications.emailContains(keyword)); Filtering Across Joins If your Customer has a nested Address, you can reach into it without any joins in your service layer: Java public static Specification<Customer> cityEquals(String city) { return (root, query, cb) -> city == null ? null : cb.equal(root.join("address").get("city"), city); } The join happens inside the Specification. Your service code stays clean. Pagination Because JpaSpecificationExecutor exposes a findAll(Specification, Pageable) overload, adding pagination is one line: Java Page<Customer> page = customerRepository.findAll(spec, PageRequest.of(0, 20, Sort.by("name"))); Mistakes I've Seen in the Wild Returning non-null predicates for null filters: This is the most common gotcha. If you forget the null check and return a valid predicate anyway, you'll silently filter out data that should be returned. Always guard at the top of the lambda. Mixing business logic into Specifications: A Specification should do one thing: produce a predicate. I've seen Specifications that log, that call services, that check permissions. Don't. Keep them pure. Creating a single "God Specification" that handles all filters: This trades the bloated repository problem for a bloated Specification problem. Small, single-purpose Specifications stay testable and reusable. A statusEquals Specification can serve your search screen, your reporting module, and your admin dashboard without any of them knowing about each other. Skipping case normalization for string searches: cb.like(root.get("name"), "%dzone%") won't match "DZone" or "DZONE." Always normalize: cb.lower(root.get("name")) paired with a lowercased input. Why This Pays Off Over Time The real dividend from Specifications shows up six months after you introduce them, when requirements change, and they always do. Adding a filter? One new static method, one .and(). Removing a filter? Delete the method and the combinator line. Reusing a filter across two features? Import the same Specification class. Unit testing a filter? Instantiate the Specification, pass a mock CriteriaBuilder, assert the predicate. No Spring context required. In complex enterprise codebases, the kind with multiple development teams, evolving product requirements, and a long maintenance tail, that kind of modularity is worth a lot more than it sounds at first. Final Thought Specifications aren't exotic. They're part of the Spring Data JPA standard library; they work with everything you already have, and they solve a problem that every team with a search screen eventually hits. If your repository is starting to look like an alphabetized index of every filter combination your users have ever requested, it's a good time to make the switch.

Most Spring Boot APIs I’ve reviewed have a security configuration that was correct three commits ago. Then somebody added a new endpoint, the security config didn’t get the matching update, and now there’s an unauthenticated path under /api/internal/ that returns a JSON dump of every active user. The team didn’t intend it; the framework didn’t catch it; the SAST tool flagged it three weeks later when the next scan ran. This article is a reference implementation. JWT authentication done correctly, rate limiting that survives distributed deployments, input validation that catches more than annotations alone, and output encoding that doesn’t break under the edge cases. Each section has the code that should be on every API by default, with the rationale for why. The Baseline Security Configuration Spring Security’s default behavior is to deny everything. That’s the right default. The configuration that follows opens up only what should be open and authenticates everything else. Java @Configuration @EnableWebSecurity @EnableMethodSecurity public class SecurityConfig { @Bean public SecurityFilterChain filterChain(HttpSecurity http, JwtDecoder jwtDecoder) throws Exception { http .csrf(csrf -> csrf.disable()) // For stateless JWT APIs .sessionManagement(s -> s.sessionCreationPolicy(STATELESS)) .authorizeHttpRequests(auth -> auth .requestMatchers("/health", "/metrics").permitAll() .requestMatchers("/api/public/**").permitAll() .anyRequest().authenticated() ) .oauth2ResourceServer(oauth2 -> oauth2 .jwt(jwt -> jwt.decoder(jwtDecoder)) ) .headers(headers -> headers .contentSecurityPolicy(csp -> csp.policyDirectives( "default-src 'self'; frame-ancestors 'none'")) .frameOptions(frame -> frame.deny()) .httpStrictTransportSecurity(hsts -> hsts .includeSubDomains(true) .maxAgeInSeconds(31536000)) ); return http.build(); } } A few specific decisions: CSRF disabled. Only because we’re stateless and using bearer tokens. If your API uses session cookies, leave CSRF enabled. The vulnerability that CSRF protects against requires session-based authentication. anyRequest().authenticated() at the end. The catch-all matters. Without it, an endpoint that doesn’t have an explicit rule defaults to permitted. With it, an endpoint without an explicit rule requires authentication. The latter is the safer default. Security headers. CSP, frame-options, HSTS. These are the headers that mitigate browser-side attacks. Even an API that never serves to a browser benefits from them, because errors might leak through a browser at some point. JWT Validation That Doesn’t Have Known Holes JWT is widely deployed and widely misconfigured. The specific mistakes that show up in scans: Accepting none algorithm. The JWT spec includes a none algorithm for unsigned tokens. If your decoder accepts it, an attacker can forge any token. Spring Security’s default JWT decoder doesn’t accept none, but if you’ve written a custom one, check. Confusing HS256 and RS256. A symmetric algorithm (HS256) means the secret key both signs and verifies. An asymmetric algorithm (RS256) means a private key signs and a public key verifies. If your service is configured to verify with a key but accepts whichever algorithm the token specifies, an attacker can sign with the public key (treating it as a symmetric secret), and your service will accept it. Pin the algorithm explicitly. Skipping signature verification. Yes, this happens. The decoder is supposed to verify; somebody disabled it during debugging; it never got re-enabled. The configuration: Java @Bean public JwtDecoder jwtDecoder(@Value("${jwt.issuer-uri}") String issuer, @Value("${jwt.audience}") String expectedAudience) { NimbusJwtDecoder decoder = NimbusJwtDecoder .withIssuerLocation(issuer) .jwsAlgorithm(SignatureAlgorithm.RS256) // Pin the algorithm .build(); OAuth2TokenValidator<Jwt> withIssuer = JwtValidators.createDefaultWithIssuer(issuer); OAuth2TokenValidator<Jwt> withAudience = new JwtClaimValidator<List<String>>( JwtClaimNames.AUD, aud -> aud != null && aud.contains(expectedAudience)); OAuth2TokenValidator<Jwt> withClockSkew = new JwtTimestampValidator(Duration.ofSeconds(30)); decoder.setJwtValidator(new DelegatingOAuth2TokenValidator<>( withIssuer, withAudience, withClockSkew)); return decoder; } JwtClaimValidator ships with Spring Security and lets you assert a predicate against any claim. (Spring Boot 3.2+ also exposes a shortcut: set spring.security.oauth2.resourceserver.jwt.audiences in application properties and the framework wires up audience validation for you.) The audience check is what stops a token meant for one service from being used against another. If your identity provider issues tokens for several services, each service should require its own audience claim. The clock skew (30 seconds) accounts for clock drift between the token issuer and your service. Tokens issued slightly in the future or expired slightly in the past are still accepted. Authorization at the Method Level Authentication is who you are. Authorization is what you can do. Spring’s method-level security is the place to express it. Java @RestController @RequestMapping("/api/orders") public class OrderController { @GetMapping("/{orderId}") @PreAuthorize("@orderAuthorization.canRead(#orderId, authentication)") public Order getOrder(@PathVariable String orderId) { return orderService.findById(orderId); } @PostMapping @PreAuthorize("hasAuthority('SCOPE_orders:write')") public Order createOrder(@Valid @RequestBody CreateOrderRequest request, Authentication authentication) { return orderService.create(request, authentication.getName()); } @DeleteMapping("/{orderId}") @PreAuthorize("@orderAuthorization.canDelete(#orderId, authentication) " + "and hasAuthority('SCOPE_orders:delete')") public void deleteOrder(@PathVariable String orderId) { orderService.delete(orderId); } } The @orderAuthorization reference is a custom bean that handles the data-dependent authorization. The reason for using a SpEL expression rather than putting the logic in the controller body: the authorization check happens before the controller method runs, which means the method body can assume authorization passed and doesn’t have to repeat the check. The custom bean: Java @Component("orderAuthorization") public class OrderAuthorization { private final OrderRepository orderRepo; public boolean canRead(String orderId, Authentication auth) { Order order = orderRepo.findById(orderId).orElse(null); if (order == null) return false; Jwt jwt = (Jwt) auth.getPrincipal(); String userId = jwt.getSubject(); // Owner can always read their own orders if (order.getOwnerId().equals(userId)) return true; // Admin role can read any order Collection<? extends GrantedAuthority> authorities = auth.getAuthorities(); return authorities.stream() .anyMatch(a -> a.getAuthority().equals("ROLE_ADMIN")); } } For most APIs, this two-layer approach (scope at the method, data check in a bean) is the right shape. Scopes are for “this caller is allowed to do this kind of thing.” Data checks are for “this caller is allowed to do this thing to this specific resource.” Input Validation Beyond Annotations Bean validation annotations (@NotNull, @Size, @Email, etc.) catch the simple cases. They don’t catch: Cross-field validation (start date before end date).Conditional validation (if status is “shipped,” tracking number is required).Value normalization (trimming whitespace, normalizing email casing).Defense against the input that’s technically valid but operationally wrong (a 1000-element array of valid email addresses in a request that should accept at most 10). The pattern that’s worked: Java public record CreateOrderRequest( @NotBlank @Size(max = 100) String customerId, @NotEmpty @Size(max = 50) List<@Valid OrderItem> items, @Size(max = 500) String notes ) { public record OrderItem( @NotBlank @Size(max = 50) String productId, @Min(1) @Max(999) int quantity, @NotNull @DecimalMin("0.00") @Digits(integer = 8, fraction = 2) BigDecimal price ) {} } Each field has a maximum. The list has a maximum. Money is BigDecimal with explicit digit limits, not double. The annotations cover the structural rules. For business rules, a separate validator: Java @Service public class OrderRequestValidator { public void validate(CreateOrderRequest request) { BigDecimal total = request.items().stream() .map(item -> item.price().multiply(BigDecimal.valueOf(item.quantity()))) .reduce(BigDecimal.ZERO, BigDecimal::add); if (total.compareTo(MAX_ORDER_VALUE) > 0) { throw new ValidationException("Order value exceeds maximum"); } // Other business validations... } } The split keeps the structural validation in the request type (where Spring auto-applies it) and the business validation in a service that the controller calls explicitly. A validation rule had set a minimum length of five characters on the last name field — a reasonable default that nobody questioned. It broke for users with single-letter last names, and it broke silently for users with no last name at all. The annotation rejected both without a meaningful error. The records existed in the source system; they just couldn't be processed. We found it not through testing but through a support ticket from an operator who couldn't explain why a specific record kept failing. The fix was a minimum of one character and explicit handling for the absent last name case. The lesson was that "reasonable default" and "correct default" aren't the same thing, and that test data built from typical cases misses the atypical ones that real populations contain. Output Encoding The output side gets less attention than the input side. It deserves equal attention. Specifically: JSON serialization that doesn’t leak. Don’t return your JPA entity directly. Return a DTO that includes only the fields the API contract specifies. The reasons: an entity has fields that aren’t in the contract (audit columns, internal references, lazy-loaded relationships) and serializing them by accident leaks information. Worse, if you ever add a field to the entity, it gets exposed without a contract change. Java public record OrderResponse( String id, String customerId, OrderStatus status, List<OrderItemResponse> items, BigDecimal total, Instant createdAt ) { public static OrderResponse from(Order order) { return new OrderResponse( order.getId(), order.getCustomerId(), order.getStatus(), order.getItems().stream().map(OrderItemResponse::from).toList(), order.calculateTotal(), order.getCreatedAt() ); } } The DTO is explicit about what crosses the wire. Adding a field to the entity doesn’t change the API surface unless somebody also updates the DTO. Error responses that don’t leak. A controller advice that catches uncaught exceptions and returns a sanitized response: Java @RestControllerAdvice public class GlobalExceptionHandler { @ExceptionHandler(Exception.class) public ResponseEntity<ErrorResponse> handleAny(Exception e) { String correlationId = UUID.randomUUID().toString(); log.error("Unhandled exception, correlationId={}", correlationId, e); return ResponseEntity .status(HttpStatus.INTERNAL_SERVER_ERROR) .body(new ErrorResponse( "INTERNAL_ERROR", "An unexpected error occurred", correlationId )); } @ExceptionHandler(MethodArgumentNotValidException.class) public ResponseEntity<ErrorResponse> handleValidation(MethodArgumentNotValidException e) { return ResponseEntity .badRequest() .body(new ErrorResponse("VALIDATION_FAILED", "Request validation failed", null)); } } The correlation ID is the link between the response and the server log. The user gets a meaningless ID, the operator can find the full error in the logs, and no internal information leaks through the response. Rate Limiting The simple in-memory bucket works for a single instance. The moment you have two instances, the rate limit per instance is double the rate limit per user. The pattern that scales: Java @Component public class RateLimitFilter extends OncePerRequestFilter { private final RedisRateLimiter limiter; @Override protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain) throws ServletException, IOException { String key = "rl:" + extractUserKey(request); RateLimitDecision decision = limiter.tryConsume(key, 100, Duration.ofMinutes(1)); response.setHeader("X-RateLimit-Limit", "100"); response.setHeader("X-RateLimit-Remaining", String.valueOf(decision.remaining())); if (!decision.allowed()) { response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value()); response.setHeader("Retry-After", String.valueOf(decision.retryAfterSeconds())); return; } chain.doFilter(request, response); } } The RedisRateLimiter uses a Redis-backed token bucket (libraries like Bucket4j with Redis support work, or Resilience4j, or you can write the Lua script yourself). The key insight is that the limiter state is shared across all instances of your service. The 429 response includes a Retry-After header. Clients that respect it back off automatically. Clients that don’t are easier to identify in your logs. For PHI APIs, rate limiting is also a security signal. A user account suddenly hitting the rate limit is anomalous. Log it, and consider an alert. The rate limit surfaced something we wouldn't have caught otherwise. A staff account on a records API started hitting the rate limit consistently during off-hours — request volume that made no operational sense for that role at that time. The account wasn't doing anything the API would reject on its own; each individual request was authorized and valid. The pattern was what was wrong. We investigated, found the credentials had been compromised, and revoked them. The rate limit didn't prevent the breach — the credentials had already been in use for some time by the time we investigated. What it did was create a signal we could act on. Without it, the access would have continued until someone noticed the data was wrong, which is a much longer detection window. For APIs handling sensitive records, the 429 log entry is as important as the 403. CORS Done Specifically CORS configuration is one of those areas where a small mistake creates a big hole. The pattern: Java @Configuration public class CorsConfig { @Bean public CorsConfigurationSource corsConfigurationSource( @Value("${cors.allowed-origins}") String allowedOrigins) { CorsConfiguration config = new CorsConfiguration(); config.setAllowedOrigins(Arrays.asList(allowedOrigins.split(","))); config.setAllowedMethods(List.of("GET", "POST", "PUT", "PATCH", "DELETE")); config.setAllowedHeaders(List.of("Authorization", "Content-Type")); config.setExposedHeaders(List.of("X-RateLimit-Remaining")); config.setAllowCredentials(true); config.setMaxAge(3600L); UrlBasedCorsConfigurationSource source = new UrlBasedCorsConfigurationSource(); source.registerCorsConfiguration("/api/**", config); return source; } } The allowed origins come from configuration, not hardcoded. The credentials flag is true (we want cookies and Authorization headers to flow), which means the origins must be specific (no wildcards). The exposed headers list is explicit because the browser hides response headers from JavaScript by default. Logging That Doesn’t Poison Itself The logging configuration matters as much as anything else. The patterns: A logging filter that scrubs sensitive fields: Java public class SensitiveDataFilter extends Filter<ILoggingEvent> { private static final Pattern SSN = Pattern.compile("\\b\\d{3}-\\d{2}-\\d{4}\\b"); private static final Pattern CARD = Pattern.compile("\\b\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}[\\s-]?\\d{4}\\b"); @Override public FilterReply decide(ILoggingEvent event) { String msg = event.getFormattedMessage(); if (SSN.matcher(msg).find() || CARD.matcher(msg).find()) { // Best-effort: scrub or drop // ... in practice, replace with a redacted version } return FilterReply.NEUTRAL; } } A regex-based scrubber is imperfect but better than nothing. The preferable approach is to never log the sensitive data in the first place, by being deliberate about what goes into log statements. Structured logging. JSON logs with consistent field names make queries possible. The correlation ID in the error response comes from the structured log entry. Without it, debugging in production is grep against unstructured text. Audit logging is separate. The audit log isn’t the application log. It’s a separate stream with separate retention, separate access controls, and a separate schema. Mixing them is the most common mistake I’ve seen. The one that consumed the most time wasn't a missing control — it was a configuration conflict. A global security config was overriding a local one. The global config had security settings it wasn't supposed to have according to company protocol; security at that level was meant to be handled locally by each service. Because the global config was authoritative in the resolution order, the local configuration's rules were silently ignored. Everything appeared to be working — requests were being accepted, responses were coming back — but the security posture in effect wasn't the one anyone had reviewed or intended. It took a long time to find it because nobody thought of looking at the global config; the assumption was that the local config was what was running. The fix was straightforward once we found it. The lesson was that configuration precedence needs to be as explicit and documented as the configuration itself — knowing what your security config says matters less than knowing which security config is active. What I’d Put On Every Spring Boot API Reduced to the essentials: JWT authentication with explicit algorithm pinning, audience validation, and issuer validation.Method-level authorization with both scope checks and data-dependent checks.Input validation with Bean Validation annotations on the request types, plus a service-layer validator for business rules.Explicit DTOs for responses, never raw entity serialization.Global exception handling that returns sanitized errors with correlation IDs.Distributed rate limiting with appropriate headers.CORS configured with specific origins and credentials.Structured logging with sensitive field scrubbing.Security headers in the filter chain (CSP, HSTS, frame-options). The reference implementation isn’t a starting point you keep static. It’s the floor. Every API should have at least this. Beyond it, the security model gets specific to the data, the threat model, and the operational environment. But this floor catches the patterns that scanners flag and that real attackers exploit. Without it, you’re not arguing about defense in depth. You’re missing the basic controls.

Most articles on Page Object Model are written by people who maintain twelve tests. This is one written by somebody who has lived inside a 2,400-test web automation suite for three years and watched it ossify, get rebuilt, and ossify again. I don’t think POM is wrong. I think the version of POM that gets taught — one class per page, methods that wrap WebElement clicks — falls apart somewhere around 300 tests. The version we run today still calls itself POM, and the page classes look like the original ones, but underneath there are three or four patterns layered on that nobody told me about when I started. This article is about those patterns. The stack: Java 17, Selenium 4.14, TestNG 7.8, Maven, running locally and on a Selenium Grid 4 in Kubernetes. About 2,400 tests, ~38 minutes wall time on 24 parallel nodes, ~6% flake rate that we are continuously fighting to keep under 8%. What the Textbook POM Gives You, and Where It Stops The textbook is fine for one page. You write a LoginPage, it has a loginAs(user, pass) method, your test calls it. You feel good. Now you have 40 pages, half of them inherit a header and footer, three have modal dialogs, one is a wizard with seven steps, and you’ve got a single BasePage class that’s 1,100 lines long and includes a method called Wait-For-Thingie-To-Be-Ready-But-Only-If-FlagX-Is-Set. The pain points I hit, in the order I hit them: Pages that have shared regions (the global header, the side nav, a footer that’s actually loaded async). If you put header methods on every page class, you get duplication. If you put them on BasePage, you get a 1,100-line god class. Pages that are really states. A “shopping cart” isn’t one page; it’s empty-cart, populated-cart, and during-checkout. The same URL, three behaviors. Wait strategies that need to be page-specific. The dashboard takes 4-7 seconds to load because it’s running 11 GraphQL queries; the settings page loads instantly. A single global Thread.sleep(5000) in BasePage is how you get a 90-minute test suite. Tests that need to set up state without going through the UI. We have a checkout test that needs the user to already have three items in their cart. Going through the UI to add three items is 14 seconds per test, times the 200 tests that need a populated cart = a lot of compute. Cross-browser differences. Chrome and Firefox behave differently around shadow DOM. A click that works in Chrome might silently no-op in Firefox 119. POM by itself doesn’t tell you where to put the workaround. The hybrid framework is the answer to those five problems. There are four patterns layered on top of textbook POM. Pattern 1: Component Classes for Shared Regions The first move is to stop pretending the header and footer belong to the page. They don’t. They are components that happen to render on the page. Java public class GlobalHeader { private final WebDriver driver; private final WebDriverWait wait; @FindBy(css = "[data-test='header-search']") private WebElement searchInput; @FindBy(css = "[data-test='header-cart-icon']") private WebElement cartIcon; @FindBy(css = "[data-test='header-cart-count']") private WebElement cartCount; public GlobalHeader(WebDriver driver) { this.driver = driver; this.wait = new WebDriverWait(driver, Duration.ofSeconds(10)); PageFactory.initElements(driver, this); } public CartPage openCart() { cartIcon.click(); return new CartPage(driver); } public int getCartItemCount() { wait.until(ExpectedConditions.visibilityOf(cartCount)); return Integer.parseInt(cartCount.getText().trim()); } public SearchResultsPage search(String query) { searchInput.clear(); searchInput.sendKeys(query); searchInput.sendKeys(Keys.ENTER); return new SearchResultsPage(driver); } } Then in any page that has the header (which is every page after login), the header is a field, not inherited behavior: Java public class HomePage { private final WebDriver driver; public final GlobalHeader header; public final GlobalFooter footer; public final SideNav nav; @FindBy(css = "[data-test='homepage-hero']") private WebElement hero; public HomePage(WebDriver driver) { this.driver = driver; this.header = new GlobalHeader(driver); this.footer = new GlobalFooter(driver); this.nav = new SideNav(driver); PageFactory.initElements(driver, this); } public boolean isHeroVisible() { return hero.isDisplayed(); } } In tests: HomePage home = new HomePage(driver); int cartCount = home.header.getCartItemCount(); The reason this scales: when the header changes, and headers always change, you fix one class. Not 40. We did the migration from BasePage containing the header methods to dedicated component classes in early 2023. It took two engineers about four days. Worth every hour. Sangeeta, who was new to the team at the time, did most of it; she said later it was the only refactor where she ended up writing fewer lines of code than she deleted. Components nest. Our CheckoutPage has a CheckoutSidebar which has a PromoCodeWidget which has its own apply/clear methods. Test code reads down the tree: checkout.sidebar.promoCode.apply("HOLIDAY20"); It reads like the actual product. That’s the test of a good POM split: does the test code map to how a user would describe what they’re doing? Pattern 2: Loadable Component for State-Aware Pages Borrowed shamelessly from Selenium’s LoadableComponent, but we ended up writing our own because Selenium’s version assumes you can get() a URL to load, which doesn’t work for SPAs and modal dialogs. Java public abstract class LoadablePage<T extends LoadablePage<T>> { protected final WebDriver driver; protected final WebDriverWait wait; protected LoadablePage(WebDriver driver) { this.driver = driver; this.wait = new WebDriverWait(driver, Duration.ofSeconds(15)); } /** Returns true once the page is loaded enough to interact with. */ protected abstract boolean isLoaded(); /** Override to return a useful error if isLoaded() times out. */ protected String loadError() { return getClass().getSimpleName() + " did not load within timeout."; } @SuppressWarnings("unchecked") public T waitUntilLoaded() { try { wait.until(d -> isLoaded()); } catch (TimeoutException e) { throw new RuntimeException(loadError() + " Current URL: " + driver.getCurrentUrl(), e); } return (T) this; } } A page extends it: Java public class DashboardPage extends LoadablePage<DashboardPage> { @FindBy(css = "[data-test='dashboard-widgets-loaded']") private WebElement loadedSentinel; public DashboardPage(WebDriver driver) { super(driver); PageFactory.initElements(driver, this); } @Override protected boolean isLoaded() { try { return loadedSentinel.isDisplayed(); } catch (NoSuchElementException | StaleElementReferenceException e) { return false; } } } The trick is the [data-test='dashboard-widgets-loaded'] element. It’s a hidden div that the application renders when all 11 GraphQL queries on the dashboard have resolved. We had to ask the frontend team to add it. They pushed back at first (“you should test the user-visible state, not internal state”) and then I showed them the 27 tests that were flaking because we were waiting on the wrong element. They added the div. This is the negotiation move that nobody writes about: getting data-test attributes added to the application is half of POM in practice. Aman on the frontend team and I had a recurring 15-minute Tuesday standup for about three months in 2023 where we went through “tests flaked, here are the elements I need stable selectors for,” and he’d merge the PR by Wednesday. That meeting did more for our flake rate than any wait-strategy refactor. Pattern 3: Test Data Builders, Not UI Setup Tests that need preconditions should not click their way through the UI to set up. They should call APIs. We have a TestDataBuilder that sits next to the page objects. It uses the same authenticated session as the test: Java public class TestDataBuilder { private final ApiClient api; private final String userId; public TestDataBuilder(ApiClient api, String userId) { this.api = api; this.userId = userId; } public CartBuilder withCart() { return new CartBuilder(); } public class CartBuilder { private final List<String> productSkus = new ArrayList<>(); private String promoCode; public CartBuilder withProduct(String sku) { productSkus.add(sku); return this; } public CartBuilder withProducts(String... skus) { productSkus.addAll(Arrays.asList(skus)); return this; } public CartBuilder withPromoCode(String code) { this.promoCode = code; return this; } public Cart build() { // POST /api/cart with the user's auth token CartResponse resp = api.post("/cart", Map.of("userId", userId, "skus", productSkus, "promo", promoCode), CartResponse.class); return new Cart(resp.cartId); } } } In the test: Java @Test public void checkoutWithFullCart() { Cart cart = data.withCart() .withProducts("SKU-1029", "SKU-3344", "SKU-4101") .withPromoCode("HOLIDAY20") .build(); CartPage page = new CartPage(driver).waitUntilLoaded(); assertEquals(3, page.getItemCount()); CheckoutPage checkout = page.proceedToCheckout(); // ... rest of test } That builder skipped the 14 seconds of UI clicks. Multiplied across 200 tests on every CI run, it cut about 47 minutes off our suite wall time. The other thing it did, which I didn’t expect, was reduce flake, because the test wasn’t fighting the UI for setup; the actual assertion ran cleaner. The pushback I got on this approach was philosophical: “you’re not testing the cart-add flow if you skip it.” Correct. We test the cart-add flow in one dedicated test. We don’t re-test it 200 times across other suites. This is the same argument as “don’t test the framework”; it just hits earlier than people expect. Pattern 4: Browser-aware action helpers When you find a Chrome-Firefox-Safari difference, you want exactly one place to put the workaround. We have an Actions helper class that wraps the most common interactions and dispatches per browser: Java public class SmartActions { private final WebDriver driver; private final BrowserType browser; public SmartActions(WebDriver driver) { this.driver = driver; this.browser = detectBrowser(driver); } public void click(WebElement el) { switch (browser) { case FIREFOX -> firefoxClick(el); case SAFARI -> safariClick(el); default -> el.click(); } } private void firefoxClick(WebElement el) { // Firefox 119 has a bug where clicks on elements with // pointer-events: none parents silently fail. JS click bypasses. if (isInsideShadowDom(el)) { ((JavascriptExecutor) driver).executeScript("arguments[0].click();", el); } else { el.click(); } } private void safariClick(WebElement el) { // Safari needs a scroll-into-view before click on long pages ((JavascriptExecutor) driver).executeScript( "arguments[0].scrollIntoView({block: 'center'});", el); try { Thread.sleep(150); } catch (InterruptedException e) {} el.click(); } } That Thread.sleep(150) in SafariClick offends every clean-code instinct in your body. It’s there because it works and the documented WebDriverWait alternatives don’t. The Safari driver has its own race condition between scroll and click that I tracked through 60 hours of debugging and ended up logging as a bug against safaridriver. They acknowledged it, and I haven’t seen a fix. Page objects use SmartActions instead of calling .click() directly: public CartPage proceedToCheckout() { actions.click(checkoutButton); return new CartPage(driver); } When a new browser quirk shows up, you fix it in one place and 2,400 tests inherit the fix. TestNG configuration that actually parallelizes Selenium’s parallelization story is fine. TestNG’s parallelization story is fine. Getting them to play nice with a remote grid took longer than I want to admit. The key insight: parallel="methods" plus thread-count in your testng.xml is necessary but not sufficient. You also need a WebDriver factory that creates a new driver per thread, and a BeforeMethod that doesn’t accidentally leak drivers across threads. Java public class DriverFactory { private static final ThreadLocal<WebDriver> DRIVER = new ThreadLocal<>(); public static WebDriver get() { if (DRIVER.get() == null) { DRIVER.set(create()); } return DRIVER.get(); } private static WebDriver create() { String browser = System.getProperty("browser", "chrome"); String gridUrl = System.getProperty("grid.url", "http://selenium-hub:4444/wd/hub"); DesiredCapabilities caps = new DesiredCapabilities(); caps.setBrowserName(browser); try { return new RemoteWebDriver(new URL(gridUrl), caps); } catch (MalformedURLException e) { throw new RuntimeException(e); } } public static void quit() { WebDriver d = DRIVER.get(); if (d != null) { d.quit(); DRIVER.remove(); } } } BaseTest: public abstract class BaseTest { protected WebDriver driver; @BeforeMethod(alwaysRun = true) public void setUp() { driver = DriverFactory.get(); driver.manage().window().setSize(new Dimension(1440, 900)); } @AfterMethod(alwaysRun = true) public void tearDown() { DriverFactory.quit(); } } The ThreadLocal matters. Without it, two parallel test threads will share a driver and corrupt each other. The first time we deployed to the grid, we had a 22% flake rate that was almost entirely shared-driver corruption. Adding ThreadLocal fixed it in an afternoon. testng.xml for the parallel run: XML <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd"> <suite name="full-regression" parallel="methods" thread-count="24"> <listeners> <listener class-name="com.example.framework.RetryListener"/> <listener class-name="com.example.framework.ScreenshotOnFailureListener"/> </listeners> <test name="regression"> <packages> <package name="com.example.tests.regression"/> </packages> </test> </suite> The retry listener handles transient failures (network blip, grid node death). One retry, no more. We had a phase where engineers were setting it to retry 3-5 times, and the suite would “pass” but actually be hiding real bugs. One retry. That’s the rule. If a test needs three retries to pass, it’s a flaky test, and we file it as a bug, not a feature. What I’d tell you to skip Two things I tried and reverted on. We spent a quarter trying to use Cucumber as the test runner because some product stakeholder wanted “BDD.” We wrote ~80 step definitions, and they slowed everything down. The page objects were now wrapped in step definitions, which made the test code less expressive, not more. The product stakeholder lost interest by Q2, and we ripped Cucumber out. If you’re considering Cucumber: be very sure the product side will actually read the .feature files. They usually don’t. We also tried to auto-generate page objects from the application’s component library. Some early @FindBy attempts used the React component names as selectors. It worked for trivial pages and broke on anything dynamic. The lesson, six months in: page objects encode test intent, not application structure. Generating them from the app’s components produces page classes that mirror the implementation, which is the opposite of what you want; a page object should outlive the implementation that backs it. What we’re working on now The current evolution is moving the heavier setup builders behind a service that the test suite calls. Right now, TestDataBuilder calls our app’s API directly; we’re abstracting that into a “test fixture service” that can also seed the database directly when the API doesn’t expose what we need. The argument for it: there are still 30 or so tests that go through the UI to set up state because the API doesn’t have an endpoint. We could either add the endpoints (the right answer; the frontend team has been saying yes for nine months) or seed via SQL (the wrong answer; the fast answer). We’re doing both, in parallel, depending on which is faster for the specific test. Anyway, that’s the framework. POM at the bottom, components above it, loadable pages on top of those, builders for setup, smart actions for browser quirks, ThreadLocal for grid parallelism. Four patterns, all of which I learned by getting them wrong first.

One of the most consequential decisions in any enterprise cloud migration is deceptively simple to state and surprisingly hard to answer: do we move the workload as-is, or do we modernize it first? Having worked through cloud migrations across dozens of enterprise customers spanning both AWS and Azure. I can tell you this question rarely has a universal answer. The right path depends on the workload, the business context, and the maturity of the team inheriting it in the cloud. What follows is the decision framework I use when guiding customers through this choice. Understanding the Two Paths Lift-and-shift (also called rehost) means moving a workload to the cloud with minimal or no code changes. You are essentially taking an on-premises virtual machine (VM), an application server, or a database and running it on cloud infrastructure instead. Tools like Azure Migrate and AWS Migration Hub (Application Migration Service, or MGN) are purpose-built for this. Modernization is a broader term that can mean refactoring an application to use cloud-native services (databases-as-a-service, managed Kubernetes, serverless functions), re-platforming to a container-based architecture, or rebuilding from scratch as a microservices application. The spectrum between these two poles includes re-platforming, for example, moving a SQL Server workload to Azure SQL Managed Instance, which preserves the database engine behavior while offloading infrastructure management. This middle path is often underrated. The Core Tension Lift-and-shift is fast and low-risk. You can move a workload in weeks, not months. Your teams do not need to rearchitect anything. Applications continue to behave exactly as they did on-premises. The downside is that you carry your technical debt into the cloud. A poorly designed, resource-hungry application that cost you money on-premises will likely cost you more in the cloud, where idle compute is billed by the hour. You also miss out on cloud-native capabilities: autoscaling, managed resilience, and pay-per-use economics. Modernization promises better long-term economics and agility. But it is expensive up front, requires skill sets your team may not yet have, and introduces real delivery risk. Projects that start as modernization efforts frequently run over time and budget. The goal of a decision framework is to apply the right approach to the right workload, not to pick a single philosophy and apply it everywhere. Five Questions That Drive the Decision 1. What Is the Business Criticality of This Workload? Tier 1: Applications that directly generate revenue or are customer-facing warrant investment in modernization, especially if they have growth potential. The engineering effort pays back through scalability, resilience, and feature velocity. Tier 3: Internal tools, reporting systems, or legacy applications used by a handful of employees are strong lift-and-shift candidates. The cost of modernizing rarely justifies the benefit. A fast triage: Ask the application owner what happens if the application is down for four hours during business hours. The answer tells you a lot about where to invest. 2. Is the Application End-of-Life or Actively Developed? If an application is on a deprecation path, to be replaced in 18 to 36 months, lift-and-shift is almost always the correct call. You want the application in the cloud for consolidation, cost, or data center exit reasons, but you do not want to invest engineering resources in something you are going to retire. Conversely, if an application is actively developed and your engineering team ships features to it regularly, modernization has a compounding return. Every sprint benefits from cloud-native capabilities. 3. What Are the Licensing and Dependency Constraints? Some applications are locked to specific operating system versions, middleware versions, or third-party components that are not certified on modern platforms. A manufacturing execution system or a financial ledger application from 2008 may have an ISV (Independent Software Vendor) support contract that explicitly requires Windows Server 2012 R2. In those cases, your choice is not lift-and-shift versus modernization. It is lift-and-shift or do nothing. Azure and AWS both offer extended security update programs for legacy OS versions, making rehost viable even for older stacks. 4. What Are the Team's Skills and Capacity? Modernization is an engineering-intensive activity. If your team is composed primarily of infrastructure engineers skilled at VM management but with limited experience in Kubernetes, Terraform, or cloud-native PaaS (Platform as a Service) services, a forced modernization will stall. Honest capacity and skills assessment matters. I have seen organizations attempt to modernize a monolithic Java application to microservices while simultaneously running a datacenter migration. Both programs suffered. A phased approach often works better: lift-and-shift first to get out of the datacenter, then modernize workloads incrementally once the team is stable on the cloud platform. 5. What Are the Unit Economics Over a Three-Year Horizon? Run the numbers. This is non-negotiable. Tools like Azure's Total Cost of Ownership (TCO) calculator or AWS Pricing Calculator can model lift-and-shift costs quickly. For modernization, you will need to factor in engineering labor costs, which are often 3x to 5x the infrastructure savings in the first year. The business case shifts in favor of modernization when: The workload has high and variable traffic (autoscaling delivers real savings)The team plans significant feature development (cloud-native accelerates delivery)The current architecture requires expensive licensed middleware that PaaS services can replace The business case favors lift-and-shift when: The workload has predictable, flat traffic (reserved instances close the cost gap)Engineering capacity is constrainedThe migration is driven by a hard datacenter exit deadline A Decision Matrix Factor Favor Lift-and-Shift Favor Modernize Business criticality Low to medium High, customer-facing Development activity Stable / end-of-life Active development Technical debt Manageable High and growing Team skill set Infrastructure-focused App dev / cloud-native capable Timeline Hard deadline Flexible Licensing constraints ISV-locked Open or replaceable Traffic pattern Flat, predictable Variable, spiky The Re-Platform Middle Path Before forcing a binary choice, evaluate re-platforming for database workloads. Moving from SQL Server on a VM to Azure SQL Managed Instance, or from Oracle to Amazon RDS, is a lift-and-shift at the application layer and a modernization at the data layer. You eliminate OS patching, get automated backups, built-in high availability, and elastic scaling without refactoring a single line of application code in most cases. This is often the highest-return migration move available to enterprise customers and is underutilized because teams think in binary terms. What I See Go Wrong The most common failure mode is scope creep driven by modernization enthusiasm. A team scopes a lift-and-shift, then someone says “while we’re at it, let’s containerize it.” Twelve months later, the application is still not in production. The second most common failure mode is lift-and-shift without right-sizing. Teams migrate on-premises VMs 1:1 to cloud VMs without analyzing actual CPU and memory utilization. Azure Migrate’s performance-based assessments and AWS Compute Optimizer exist for exactly this reason. A VM provisioned at 16 cores on-premises is often running at 8% CPU utilization. Moving it as-is is leaving money on the table. Both mistakes are avoidable with a disciplined assessment phase before migration execution begins. Putting the Framework Into Practice In a typical enterprise migration engagement, I recommend the following sequencing: Discover and classify: Run an agentless discovery (Azure Migrate or AWS MGN) to inventory all workloads. Classify each by tier, development activity, and licensing constraints.Apply the decision matrix: Score each workload and assign a migration strategy: rehost, re-platform, or modernize.Sequence by risk: Start migrations with lower-criticality, lower-complexity workloads to build team confidence on the target platform.Right-size before you migrate: Use performance data to set cloud VM sizes. Do not replicate on-premises provisioning patterns.Modernize in waves: Once lift-and-shift workloads are stable in the cloud, identify the top candidates for modernization based on business value and team readiness Closing Thoughts There is no universally correct answer between lift-and-shift and modernize. The decision is contextual, and applying the wrong strategy to a workload modernizing something that should have been retired, or lifting-and-shifting something that needed to be rebuilt creates costs that compound over time. The framework above does not eliminate judgment. It structures the judgment so it is applied consistently, documented, and defensible to stakeholders who will inevitably ask why you chose the path you did.

The default playbook for adapting a foundation model looks like this: grab a pre-trained model, collect labeled data, fine-tune, deploy. It works often enough that teams rarely question it, but fine-tuning is one of six adaptation strategies. Picking the wrong one costs you weeks of engineering time and thousands in compute - sometimes for accuracy you'll never get back. The real question isn't how to fine-tune. It's whether you should. I've spent the last several years working on model adaptation across domains where labeled data is scarce and off-the-shelf models fall flat. This article lays out the decision framework I wish I'd had when I started: how to choose the right strategy based on how much labeled data you have, how far your domain sits from the model's pre-training distribution, and how much you can spend. The Six Strategies There are six ways to adapt a foundation model to a new task. I'm listing them in order of increasing investment: 1. Prompt Engineering Modify the input to the model by writing instructions, providing context, and being specific about output format. This needs zero labeled data. 2. Few-Shot/In-Context Learning (ICL) Include 3–20 labeled examples directly in the prompt. Still no training. The model picks up the pattern from examples at inference time. 3. Retrieval-Augmented Generation (RAG) Give the model access to your data at inference time by retrieving relevant documents and injecting them into the prompt. No training required. The model stays the same, and gets what it needs to know at inference time. 4. Fine-Tuning Update model weights on your labeled dataset. Requires hundreds to thousands of labeled examples. The model permanently learns your task. Modern parameter-efficient methods like LoRA and QLoRA have made this dramatically cheaper — you can fine-tune a 7B-parameter model on a single GPU by updating less than 1% of the weights. 5. Domain-Specific Pre-Training (Self-Supervised Learning) Take a pre-trained foundation model and continue pre-training it on a large body of unlabeled domain data using self-supervised objectives. You're keeping everything the model already knows — general visual features, language structure, spatial reasoning — and teaching it what your domain looks like on top of that. Then you fine-tune on a much smaller labeled set - because SSL teaches the model what your domain looks like, but not what you want it to do. Self-supervised objectives have no notion of your labels or your task; you still need fine-tuning to map those learned representations to actual decisions. The key here is that you start from a pre-trained model and not from scratch. 6. Train From Scratch Initialize a model with random weights and train it entirely on your own data. The model starts with knowing nothing. You teach it everything from the ground up. Maximum control, maximum cost. This is rarely justified unless your domain is so unique that no existing pre-trained model carries useful knowledge (novel modalities, proprietary data formats) or regulatory/IP constraints prevent using pre-trained weights. Most teams jump straight to Strategy 4, and that is costly. The Decision Framework Walk through these questions in order, and stop at the first one that meets your bar. 1. Can prompting meet your accuracy bar? Yes: Prompt Engineering. Ship it and iterate.No: Keep going. 2. Do you have 5–50 high-quality examples? Yes: Few-Shot/ICL. Prototype first, commit to training later.No: Keep going. 3. Why is the model getting it wrong? It doesn't have the right information (e.g., your docs, your product catalog, your policies): RAG. Keep knowledge external and updatable.It has the information but doesn't use it correctly (e.g., wrong format, bad reasoning, inconsistent outputs): Keep going. 4. Is your domain close to the model's pre-training data? Yes: Fine-Tune (LoRA). Update the weights on your labeled set.No: One more question. 5. Do you have large unlabeled domain data? Yes: Domain Pre-training (SSL) + Fine-Tune. Teach the model your world, then your task.No: Train From Scratch. (Are you sure? Try a different base model first.) The question most teams skip is the third: knowledge vs. behavior. And the question after that, domain distance, is what separates a quick fine-tune from a multi-week pre-training investment. If your data looks nothing like what the model was pre-trained on, fine-tuning a small labeled set won't close the gap. When to Use Which Prompt Engineering Your task is well-defined and the model already "knows" the domainYou need to ship today, not next monthYour accuracy bar is 80%, not 95% Anti-pattern: Teams that spend three weeks prompt-engineering a task that needs fine-tuning. If you're past version 15 of your prompt and still not hitting your bar, stop. You likely need additional signals in your data. In-Context Learning/Few Shot You have 5–50 gold-standard examplesYour task has clear input-output patternsYou want to prototype before committing to training Anti-pattern: Treating ICL as a permanent solution. It works for prototyping, but at scale you're paying inference cost for those examples on every single call. If you're running 10K requests/day with 20-shot prompts, you're burning tokens on examples that could be baked into the weights. RAG The model knows how to do the task but doesn't have the informationYour knowledge base changes frequently (product catalogs, policies, documentation)You need answers grounded in specific sources with citationsFine-tuning would bake in facts that go stale RAG is the strategy most teams overlook when they reach for fine-tuning. If your model is generating plausible but wrong answers about your company's products, the problem isn't the model's behavior but rather the model's knowledge. Fine-tuning on your docs will help temporarily, but the moment those docs change, you'll need to retrain. RAG keeps knowledge external and updatable. Think of this as you knowing accounting, but to help your client, you need some account information and numbers from them. Anti-pattern: Fine-tuning on your knowledge base to teach the model facts. You're burning compute to memorize information that may be frequently updated. Use RAG for knowledge, fine-tuning for behavior. Fine-Tuning You have 500–10K labeled examplesYour domain is close to the model's pre-training distributionYou need consistent quality at scale that you can reproduce Fine-tuning doesn't always mean updating every weight of the model. Parameter-efficient methods like LoRA (Low-Rank Adaptation) let you fine-tune by updating small adapter matrices. This cuts GPU memory by 60–80% and training time proportionally. If you're fine-tuning in 2025+, you should be using LoRA or QLoRA unless you have a specific reason not to. Anti-pattern: Fine-tuning on noisy labels. If your labeled data was produced by labelers who disagreed with each other 30% of the time, you're training on noise. A clean set of 500 examples will outperform a noisy 5,000. Domain-Specific Pre-Training Your domain looks different from the checkpoint/model you have chosen as your foundationYou have lots of unlabeled domain data (10K+ samples)Labeled data is expensive or hard to get (medical, legal, scientific, industrial)Fine-tuning alone plateaus well below your target accuracy This is the most underused strategy. People confuse it with training from scratch, but it is different. SSL starts from a pre-trained model with some existing knowledge, and you are leveraging that knowledge to build on top and extend it. For example, if you are working with pathology slides or some sensor data, the pre-trained model has no idea about those domains, but it still has knowledge about edges, shapes, textures, etc that is valuable to leverage. Once you teach the model your world, you can fine-tune on a small labeled set to teach it your specific task. For example, detecting regions of a slide that are cancerous. Train From Scratch You have millions of samplesYour domain is genuinely unique (novel modality/proprietary data format)No pre-trained model carries useful representations for your input typeYou need full control over architecture and training dynamicsRegulatory or IP constraints prevent using pre-trained weights When you train from scratch, you start with random weights. The model knows nothing. Every feature — from low-level patterns up to high-level abstractions — must be learned entirely from your data. With SSL, you inherit all of that for free from the pre-trained model and only teach the domain-specific parts. That's why the data requirements are so different: SSL needs 10K–100K unlabeled samples to bridge the domain gap; training from scratch needs millions to learn everything a foundation model already knows plus your domain. Anti-pattern: Training from scratch because "we want to own the model". Unless you have a concrete technical or legal reason or your data is a genuinely novel modality that no pre-trained model has ever seen, you're throwing away billions of tokens of pre-training for no benefit. Try SSL first. The Cost-Accuracy Tradeoff Here are some considerations as a reference while you choose which strategy works for your use-case: Strategy Labeled Data Needed relative/hypothetical Compute Cost (for comparison) Time to Deploy Typical Accuracy Gain Prompt Engineering 0 ~$0 Hours Baseline Few-Shot / ICL 5–50 ~$0 (inference cost) Hours +5–15% RAG 0 (needs knowledge base) ~$0 (infra + inference) Days +10–20% (knowledge tasks) Fine-Tuning (LoRA) 500–10K $100–$5K Days to weeks +10–25% Domain Pretrain + Fine-Tune 500 labeled + 10K unlabeled $1K–$20K Weeks +15–35% Train From Scratch 100K+ $10K–$500K+ Months Variable The sweet spot for most teams is somewhere between fine-tuning and domain pre-training, but the decision should be driven by the domain gap - how different your data is from what the model has seen already. Collecting more labeled data is not always the solution, and can be counter-productive. A good test for this is to plot the accuracy vs. data size. A plot that plateaus below your accuracy target shows you that more data is not going to bridge the gap. What the curve tells you: Curve still climbing at your largest subset → collect more labels. The model is still learning.Early plateau → the bottleneck is representation quality, not label quantity. Try domain pre-training or a different base model.High variance between runs at the same size → your labels are noisy or inconsistent. Fix labeling quality before scaling quantity.Train accuracy >> validation accuracy (overfitting) → insufficient data diversity. Your labeled set doesn't cover the distribution. Collect more varied examples, not just more examples. The Bottom Line The most expensive mistake in model adaptation isn't picking the wrong hyperparameters, but rather picking the wrong strategy entirely. Before you spin up a training job, walk the decision tree. Most of the time, the answer is simpler and cheaper than you think.