Deployment

Language models become much more useful when they can answer questions about information they were never trained on, including your internal documentation, product manuals, policies, and other proprietary data. Prompting alone cannot solve this, because the model simply does not have access to that knowledge. Retrieval-Augmented Generation, or RAG, is the most common way to bridge that gap. Spring AI comes with solid support for building RAG systems. It has been almost three years since Spring AI showed up, and in that time it has grown from an experimental member of the Spring portfolio into a mature layer over chat models, embedding models, vector stores, and the plumbing that sits between them, which happen to be exactly the pieces a RAG system needs. In this article, we build a small but complete RAG service with Spring AI 2.0. The application reads a set of documents into a PostgreSQL vector store, retrieves the fragments that are relevant to a user question, and lets Anthropic's Claude put together the answer based on those fragments. Everything runs from a standard Spring Boot project, and every step can be reproduced on macOS, Windows, or Linux. The full project is available on GitHub. If you just want to see the finished result, or you would rather skip the step-by-step build below, you can clone the repository and run it as it is. Everyone else can follow along and generate this project from scratch. The prompts themselves are kept deliberately simple. You can tune retrieval and prompts forever; here we care about the architecture and how the pieces fit together in Spring. Approach RAG is not really a single feature. It is more of a small pipeline, and the code below makes a lot more sense once its parts have names. Embedding: a vector of numbers that captures the meaning of a piece of text. Texts that mean similar things end up with vectors that are close to each other.Embedding model: the model that computes these embeddings. It is a different model from the chat model, and it has a different job.Vector store: a database that keeps text fragments together with their embeddings and can answer the question, "which stored fragments are closest in meaning to this query?"Chunking: documents are too large to embed and retrieve as a whole, so we split them into smaller fragments (chunks) before storing them.Similarity search: we embed the user question and fetch the top-k closest chunks from the store.Augmentation: we append the retrieved chunks to the user question before sending it to the chat model, so the model answers from the context we provided instead of from its training data. One thing here is worth calling out, because it shapes the whole setup of the project: the LLM model used in chat and the embedding model are two separate choices. As of today, Anthropic offers LLM models but no embedding API, so a Claude-based RAG system always has to pair Claude with an embedding model from somewhere else. Rather than bringing in a second cloud provider and a second API key, this project computes embeddings locally (inside the JVM), using Spring AI's ONNX transformers module and the well-known all-MiniLM-L6-v2 sentence transformer. It is free and fast enough for this, and it keeps everything on one API key. In our scenario, the service is an internal assistant for a fictional company called Nimbusfield Systems, and it answers employee questions based on the company handbook. The company and the handbook are fictional on purpose. Claude cannot possibly know about it, which makes it easy to verify that the answers really come from our documents and not from the model's own memory. We build this in three steps: Expose a /ask endpoint backed by Claude, with no retrieval, and show that the model cannot answer handbook questions.Ingest the handbook into PGvector at application startup: read, chunk, embed, and store.Attach Spring AI's QuestionAnswerAdvisor to the same ChatClient and ask again. Prerequisites Java 21Maven 3.9.x (the Maven wrapper included in generated projects works too)Spring Boot 4.0.xSpring AI 2.0.0Docker Desktop (macOS/Windows) or Docker Engine (Linux), used only to run PostgreSQL. A project skeleton can be generated at start.spring.io by selecting Web, Anthropic Claude, PGvector Vector Store, and Docker Compose Support. The remaining Spring AI modules are added manually below. The Claude API Key Sign in (or sign up) at the Anthropic Console, open Settings, then API Keys, and create a new key. New accounts may need a small prepaid credit before the API accepts requests, but the runs in this article cost only a few cents. The key is shown only once, so store it right away as an environment variable. If you would rather not spend anything at all, you can still follow along and read through the steps without running the calls yourself. macOS/Linux: export ANTHROPIC_API_KEY=sk-ant-... Windows (PowerShell, persists across sessions after reopening the terminal): setx ANTHROPIC_API_KEY "sk-ant-..." Solution Dependencies With the Spring AI BOM in place, there is no need to repeat versions on the individual artifacts. Initializr expresses the BOM's own version as a property rather than a hardcoded literal, so there is a single place to bump it later: XML <properties> <java.version>21</java.version> <spring-ai.version>2.0.0</spring-ai.version> </properties> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-bom</artifactId> <version>${spring-ai.version}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> A common source of confusion is that start.spring.io has no dependency literally named "Spring AI." Each provider- or store-specific starter (Anthropic Claude, PGvector Vector Database, and so on) is itself a Spring AI module, and picking one transitively pulls in the framework's core classes. (like ChatClient, VectorStore, etc.) Selecting any one of them is also what makes Initializr add the spring-ai-bom as shown above to the generated pom.xml for you. The BOM itself is never a separate item you tick on the Initializr dependency screen. The application needs six Spring AI modules on top of the web starter, each one with a single responsibility. XML <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webmvc</artifactId> </dependency> <!-- Chat model: Anthropic Claude --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-anthropic</artifactId> </dependency> <!-- Embedding model: local ONNX sentence transformer --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-model-transformers</artifactId> </dependency> <!-- Vector store: PostgreSQL + pgvector --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-starter-vector-store-pgvector</artifactId> </dependency> <!-- RAG advisor --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-vector-store-advisor</artifactId> </dependency> <!-- Document reading (PDF, Word, Markdown, HTML, and more) --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-tika-document-reader</artifactId> </dependency> <!-- Starts the database container on application startup --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> <!-- Docker Compose service connections for Spring AI vector stores --> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> </dependencies> Two models are referenced from the code here. One is the chat model, Claude, which is served from the Anthropic API. The other is the embedding model, which runs locally, right inside the application. We will look at that local embedding model in the next section. The Embedding Model By default, the transformers starter fetches tokenizer.json and model.onnx from Spring AI's own GitHub repository the first time the application starts and then caches them locally. In practice, this default setup is a bit fragile. raw.githubusercontent.com may rate-limit unauthenticated requests, and model.onnx (which is roughly 90 MB) is stored via Git LFS, whose bandwidth quota can run out independently of the ordinary rate limit. When that happens, the endpoint serves the small LFS pointer stub instead of the binary, with a normal-looking HTTP 200, and the failure only shows up later as a cryptic ONNX Runtime protobuf-parsing error rather than a clear download error. The fix is to bundle both files with the application instead of fetching them at startup. So we download them once: Shell mkdir -p src/main/resources/onnx/all-MiniLM-L6-v2 curl -fL -o src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json \ https://raw.githubusercontent.com/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/tokenizer.json curl -fL --http1.1 -o src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx \ https://media.githubusercontent.com/media/spring-projects/spring-ai/main/models/spring-ai-transformers/src/main/resources/onnx/all-MiniLM-L6-v2/model.onnx Then we point the embedding model at these local files in our application.properties, overriding the GitHub-backed defaults: Properties files spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json With these two properties set, the application never touches the network for the embedding model, neither on the first run nor on any run after it. The Database The pgvector team publishes a PostgreSQL image with the extension already installed. A compose.yaml in the project root is all we need: YAML services: pgvector: image: "pgvector/pgvector:pg17" environment: - "POSTGRES_DB=nimbusfield" - "POSTGRES_USER=nimbusfield" - "POSTGRES_PASSWORD=nimbusfield" labels: - "org.springframework.boot.service-connection=postgres" ports: - "5432" The labels entry is important. Spring Boot's Docker Compose support auto-detects connection details by matching the image name against a list of well-known images. Plain Postgres is on that list, but pgvector is not, since it is a third-party image. The label tells Spring Boot to treat this container as if it were the official Postgres image, and that is what actually makes the automatic connection wiring work. If we omit it, the container still starts, but Spring Boot never creates a ConnectionDetails bean for it, so the run fails with a connection error rather than falling back gracefully. Because spring-boot-docker-compose is on the classpath, running the application starts the container automatically and injects the connection details. This works the same way on macOS and Windows, as long as Docker Desktop is running. Anyone who prefers to manage the container manually can run the same image with docker run -p 5432:5432 .. and set the datasource properties explicitly. Configuration The complete application.properties, now including the embedding model overrides shown earlier: Properties files spring.ai.anthropic.api-key=${ANTHROPIC_API_KEY} spring.ai.anthropic.chat.model=claude-sonnet-5 spring.ai.anthropic.chat.max-tokens=1024 spring.ai.embedding.transformer.onnx.model-uri=classpath:/onnx/all-MiniLM-L6-v2/model.onnx spring.ai.embedding.transformer.tokenizer.uri=classpath:/onnx/all-MiniLM-L6-v2/tokenizer.json spring.ai.vectorstore.pgvector.initialize-schema=true spring.ai.vectorstore.pgvector.dimensions=384 spring.ai.vectorstore.pgvector.index-type=HNSW spring.ai.vectorstore.pgvector.distance-type=COSINE_DISTANCE logging.level.org.springframework.ai.chat.client.advisor=DEBUG Four details matter here. First, max-tokens is mandatory for the Anthropic API, which caps every response explicitly. Spring AI provides a default, but it is better stated than left implied. Second, the two spring.ai.embedding.transformer.* properties point the embedding model at the local files we bundled in the previous section, instead of Spring AI's own GitHub-backed defaults. See "The Embedding Model" above for why this matters. Third, initialize-schema=true enables the automatic creation of the vector-store table and the required extensions. (Since Spring AI 1.0, this no longer happens silently by default.) Fourth, dimensions=384 must match the embedding model. all-MiniLM-L6-v2 produces 384-dimensional vectors. If the embedding model changes later, the table has to be recreated, because the column type is vector(384). The Documents Two short Markdown files under src/main/resources/docs play the role of the company handbook. remote-work-policy.md Markdown # Nimbusfield Systems Remote Work Policy Employees may work remotely up to three days per week. Remote days must be registered in the portal by Thursday of the preceding week. Working from abroad is permitted for a maximum of 30 calendar days per year and requires prior approval from both the line manager and the People team. travel-expenses.md: Markdown # Nimbusfield Systems Travel and Expenses The daily meal allowance for business trips is 65 EUR in Europe and 80 USD elsewhere. Taxi rides are reimbursed only between airports, hotels, and client sites. Flights longer than six hours may be booked in premium economy. All expense reports are due within 15 working days after the trip via the portal. Thanks to the Tika reader used below, dropping PDFs or Word documents into the same folder works without any code changes. Step 1: Chat Without Retrieval We start with a service that wraps a ChatClient, built once from the auto-configured builder: Java @Service public class AssistantService { private final ChatClient chatClient; public AssistantService(ChatClient.Builder builder) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .build(); } public String ask(String question) { return chatClient.prompt() .user(question) .call() .content(); } } And a controller associated with it: Java @RestController public class AssistantController { private final AssistantService assistantService; public AssistantController(AssistantService assistantService) { this.assistantService = assistantService; } @GetMapping("/ask") public ResponseEntity<String> ask(@RequestParam("question") String question) { return ResponseEntity.ok(assistantService.ask(question)); } } Start the application (./mvnw spring-boot:run on macOS/Linux, mvnw.cmd spring-boot:run on Windows) and ask it a handbook question: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The response, as we might expect, is: I don't have that information in my available knowledge base. Nimbusfield Systems' specific travel and expense policy—including per diem rates for European business trips—isn't something I can confirm accurately. To get the correct figure, please check: The company's Travel & Expense Policy document (likely on the intranet/HR portal)Your Finance or HR department directlyYour manager, if travel budgets are pre-approved per trip Would you like help with anything else I can assist with more reliably? This gives us a baseline. The model behaves correctly given what it knows, which is nothing at all about this company. Step 2: The Ingestion Pipeline Ingestion follows Spring AI's extract, transform, load structure: a DocumentReader extracts the text, a TextSplitter chunks it, and the VectorStore embeds and stores the chunks. The embedding call happens implicitly inside vectorStore.add() call. The auto-configured TransformersEmbeddingModel is wired into the PgVectorStore and each chunk is embedded into the table. Java @Component public class HandbookIngestion implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(HandbookIngestion.class); private final VectorStore vectorStore; private final JdbcTemplate jdbcTemplate; private final Resource[] handbook; public HandbookIngestion(VectorStore vectorStore, JdbcTemplate jdbcTemplate, @Value("classpath:docs/*.md") Resource[] handbook) { this.vectorStore = vectorStore; this.jdbcTemplate = jdbcTemplate; this.handbook = handbook; } @Override public void run(ApplicationArguments args) { Integer count = jdbcTemplate.queryForObject( "select count(*) from vector_store", Integer.class); if (count != null && count > 0) { log.info("Vector store already contains {} chunks, skipping ingestion", count); return; } TokenTextSplitter splitter = TokenTextSplitter.builder() .withChunkSize(300) .build(); for (Resource resource : handbook) { List<Document> documents = new TikaDocumentReader(resource).get(); documents.forEach(doc -> doc.getMetadata().put("source", resource.getFilename())); List<Document> chunks = splitter.apply(documents); vectorStore.add(chunks); log.info("Ingested {} chunks from {}", chunks.size(), resource.getFilename()); } } } The count check makes ingestion idempotent, so restarting the application does not duplicate every chunk. And the source metadata attached to each chunk enables filtered searches later, for instance restricting retrieval to a single document. That same idempotency check has a practical downside worth pointing out. Once the vector store has data, restarting the application will not pick up edits to the handbook files, since the count check short-circuits before the splitter ever runs. To force a clean re-ingestion, for instance after changing a handbook document, tear down the container together with its data volume, not just the container: docker compose down -v The chunk size of 300 tokens is generous for documents this small. The splitter's default of 800 is aimed at larger, real-world content. Chunking is the least exciting and yet the most important knob in a RAG system: chunks that are too large dilute the similarity signals, while chunks that are too small lose their context. It is worth experimenting here: try a few different chunk sizes and see how the system behaves. Just remember to run docker compose down -v between runs, so the vector store is rebuilt from scratch each time. Step 3: Attaching the Retrieval Advisor Now we come back to the plain AssistantService from Step 1 and upgrade it, rather than writing something new. The ChatClient wiring we built earlier stays and what changes is what gets attached to it. Spring AI models the cross-cutting concerns around a chat call as "advisors", which are conceptually close to interceptors. The QuestionAnswerAdvisor embeds the incoming user question, runs a similarity search against the vector store, and appends the retrieved chunks to the prompt before it reaches Claude. Enabling RAG is therefore a change to how the ChatClient is constructed, not to how the request is handled: Java public AssistantService(ChatClient.Builder builder, VectorStore vectorStore) { this.chatClient = builder .defaultSystem(""" You are the internal assistant of Nimbusfield Systems. Answer employee questions precisely and briefly. If you do not know the answer, say so. """) .defaultAdvisors( QuestionAnswerAdvisor.builder(vectorStore) .searchRequest(SearchRequest.builder() .topK(4) .similarityThreshold(0.5) .build()) .build(), new SimpleLoggerAdvisor()) .build(); } topK(4) retrieves at most four chunks per question, and similarityThreshold(0.5) discards weak matches, so an entirely unrelated question augments the prompt with nothing rather than with noise. The SimpleLoggerAdvisor, combined with the DEBUG logging property we set earlier, prints the fully augmented prompt. This is the single most useful debugging tool while tuning retrieval, because it shows exactly what Claude was given. We restart and repeat the same request: http://localhost:8080/ask?question=What is the daily meal allowance for business trips in Europe? The daily meal allowance for business trips in Europe is 65 EUR. Same model, same question, and this time a precise answer grounded in the retrieved handbook chunk instead of a generic deflection. The debug log confirms what is going on behind the scenes: the user question arrives at Claude wrapped in a prompt that contains the retrieved handbook fragments as context. Going Further The default behavior of QuestionAnswerAdvisor is usable, but there are two refinements worth implementing if you want to take this pattern further. The first one concerns grounding. Even with retrieved context, the model may fall back on its general knowledge when the context does not actually contain the answer. The advisor accepts a custom PromptTemplate that controls how the question and the context are merged, and this is the place to enforce stricter behavior. The template must contain the query and question_answer_context placeholders: Java PromptTemplate strictTemplate = PromptTemplate.builder() .template(""" {query} Answer strictly based on the context below. If the context does not contain the answer, reply exactly: "This is not covered by the handbook." --------------------- {question_answer_context} --------------------- """) .build(); QuestionAnswerAdvisor advisor = QuestionAnswerAdvisor.builder(vectorStore) .promptTemplate(strictTemplate) .build(); Asking about, say, the parental leave policy (which is absent from our two files) now produces the fixed refusal instead of an invention. If people are going to rely on it, you want this on. The second refinement could be structured output, and it composes cleanly with retrieval. Declaring a record and calling .entity() instead of .content() gives back a typed object, with Spring AI instructing the model to respond in the matching JSON schema: Java public record HandbookAnswer(String answer, String sourceHint, boolean coveredByHandbook) { } public HandbookAnswer askStructured(String question) { return chatClient.prompt() .user(question) .call() .entity(HandbookAnswer.class); } A last note on the embedding choice. A local MiniLM model is not the strongest embedding model available, and for a large multilingual corpus a hosted embedding API or a bigger ONNX model would retrieve better. This choice is easy to reverse: EmbeddingModel is an interface, swapping the implementation is a matter of a dependency and a property, and the only hard constraint is the one mentioned earlier: the vector dimensions in PGvector have to match whatever the embedding model produces. Conclusion In this article, we built the RAG flow step by step. We started with a plain chat endpoint that could not answer anything about the Nimbusfield handbook, because Claude had never seen it. We then ingested that handbook into PGvector, embedding each chunk locally, and attached Spring AI's QuestionAnswerAdvisor to the same client. That single change was enough to turn a generic model into a service that answers from your own documents. After that, we talked about how we can tighten the grounding, so the model says it does not know when the context has no answer, and pulled the response straight into a typed Java record. If you want to take it further, clone the project, point it at your own documents, apply further the techniques we discussed in the Going Further section, play with different chunk sizes, retrieval settings, and prompts to see how the answers change. The Spring AI documentation goes deeper into advisors, vector stores, and retrieval configuration. The complete, runnable project is available on GitHub.

By Murat Balkan CORE

My first attempt to deploy a Spring Boot microservice on AWS Fargate didn’t fail loudly. It failed quietly — in a loop. ECS kept launching tasks, the Application Load Balancer kept marking them unhealthy, and the service never stabilized. The logs looked fine, the container looked fine, but the ALB replaced every task within seconds. The root cause was painfully simple: Spring Boot needed 45 seconds to start, and my ALB health‑check timeout was 5 seconds. The tasks never had a chance. That night changed how I build and deploy microservices. It forced me to rethink startup behavior, JVM sizing, networking, task definitions, and the entire CI/CD pipeline. This article is the guide I wish I had before that incident — a practitioner’s walkthrough of deploying a production‑ready Spring Boot service on AWS Fargate, with real artifacts and the details that matter when things go wrong. The Architecture That Finally Worked Once the health‑check issue was fixed, the architecture settled into a predictable, cloud‑native flow: Developers push code to GitHubGitHub Actions builds the JARDocker image is built and pushed to Amazon ECRECS service runs AWS Fargate tasksTraffic enters through an Application Load BalancerTasks run in private subnetsConfiguration comes from Parameter Store and Secrets ManagerLogs and metrics flow to CloudWatch It’s the standard modern microservice pipeline — but the difference between “standard” and “production‑ready” is in the details. The Spring Boot Service The microservice itself was simple — a REST API with a few endpoints. The real complexity wasn’t the controller logic; it was everything around it: startup time, health checks, configuration management, and container behavior under load. A Dockerfile Built for Production My first Dockerfile looked like the one many tutorials start with: a single‑stage build running as root with no JVM tuning. It worked locally but failed under real load. Fargate tasks with default JVM heap sizing inside a 2GB container are a classic OOM story. Here’s the hardened version that finally stabilized deployments: Dockerfile FROM eclipse-temurin:21-jre # Create non-root user RUN useradd -u 1001 springuser WORKDIR /app # Layer extraction for faster builds COPY target/*.jar app.jar # JVM tuning for Fargate ENV JAVA_OPTS="\ -XX:MaxRAMPercentage=75 \ -XX:+UseContainerSupport \ -XX:+ExitOnOutOfMemoryError \ " USER springuser ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"] This eliminated the OOMKilled events I saw on 2GB tasks and made startup time predictable. Pushing to Amazon ECR With Real Commands The first time I wrote down my ECR commands, they were placeholders. In production, they need to be exact: C aws ecr get-login-password --region us-east-1 \ | docker login --username AWS --password-stdin <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 docker push \ <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3 Immutable semantic version tags make rollbacks predictable and prevent “latest‑tag roulette.” The ECS Task Definition That Actually Runs in Production A real Fargate deployment lives or dies by its task definition. Here’s the JSON I use today — including secrets pulled from Parameter Store and Secrets Manager: JSON { "family": "employee-service", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "512", "memory": "1024", "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/employeeServiceRole", "containerDefinitions": [ { "name": "employee-service", "image": "<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/employee-service:1.0.3", "portMappings": [ { "containerPort": 8080, "protocol": "tcp" } ], "secrets": [ { "name": "DB_PASSWORD", "valueFrom": "arn:aws:ssm:us-east-1:<ACCOUNT_ID>:parameter/db/password" }, { "name": "API_KEY", "valueFrom": "arn:aws:secretsmanager:us-east-1:<ACCOUNT_ID>:secret:thirdparty/api" } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/employee-service", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "ecs" } } } ] } The ALB Health Check That Stopped the Outage My outage happened because the ALB was impatient. Here’s the configuration that finally stabilized deployments: settingvalue Path /actuator/health Interval 20 seconds Timeout 10 seconds Healthy threshold 3 Unhealthy threshold 3 Spring Boot startup time + ALB patience = stable deployments. Why Fargate Tasks Belong in Private Subnets Early on, I deployed tasks in public subnets because it felt simpler. It wasn’t. Public IPs meant the containers were directly reachable from the internet — port scans, bot traffic, and noisy logs. Moving tasks to private subnets solved several problems at once: Reduced Attack Surface No public IPs. No direct inbound traffic. Only the ALB can reach the tasks. A Single Secure Entry Point The ALB handles TLS termination, redirects HTTP→HTTPS, performs health checks, and integrates with WAF. Clients never bypass it. Cleaner Security Groups ALB SG: inbound 443 from the internetTask SG: inbound only from ALB SG Nothing else touches the containers. Compliance Alignment PCI, SOC 2, HIPAA — all prefer minimizing public exposure. Controlled Outbound Access Tasks use a NAT Gateway for outbound calls (updates, third‑party APIs) without exposing themselves. Better Scalability ALB target groups automatically track tasks across AZs as ECS scales. The architecture becomes simple and predictable: Internet → ALB (public subnets) → Fargate tasks (private subnets) It’s quieter, safer, and easier to operate. The GitHub Actions Workflow That Deploys Automatically Here’s the pipeline that builds, tests, pushes, and deploys the service: YAML name: Deploy to Fargate on: push: branches: ["main"] jobs: build-deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up JDK uses: actions/setup-java@v4 with: java-version: "21" - name: Build JAR run: mvn -B clean package - name: Login to ECR uses: aws-actions/amazon-ecr-login@v2 - name: Build and Push Image run: | docker build -t employee-service:1.0.3 . docker tag employee-service:1.0.3 ${{ env.ECR_REGISTRY }/employee-service:1.0.3 docker push ${{ env.ECR_REGISTRY }/employee-service:1.0.3 - name: Deploy ECS Service uses: aws-actions/amazon-ecs-deploy-task-definition@v2 with: task-definition: ecs-task.json service: employee-service cluster: prod-cluster Auto Scaling With Real Target Tracking JSON Target tracking is the simplest and most reliable scaling strategy for Fargate: JSON { "TargetValue": 50.0, "PredefinedMetricSpecification": { "PredefinedMetricType": "ECSServiceAverageCPUUtilization" }, "ScaleOutCooldown": 30, "ScaleInCooldown": 60 } I use 50% as the target because it balances cost and responsiveness. What I Learned Every failure taught me something: ALB timeouts taught me to respect startup timeOOMKilled tasks taught me to tune the JVMPublic subnets taught me to isolate workloadsManual deployments taught me to automate everything AWS Fargate really does deliver on its promise — no servers to manage, automatic scaling, and clean integration with ECS — but only after you learn the hard parts. If you’re deploying Spring Boot on Fargate, I hope you learn those lessons from this article instead of from your own outage.

Six months ago, building a RAG pipeline meant a full week of plumbing: an embedding job here, a vector store there, a retriever glued on with duct tape, and an orchestration layer that broke every time you touched it. I've built enough of these the hard way — hand-rolled vector search, custom chunking scripts, the works — to know exactly how much pain that "week" usually hides. Last week, I rebuilt the same thing on Azure AI Foundry. It took an afternoon. Not because the underlying problem got easier — grounding an LLM in your own data is still genuinely hard — but because Microsoft finally killed most of the integration tax that used to eat the first sprint of every RAG project. Here's what actually happened, warts included. The Old Way Was a Trap If you've built RAG before, you know the pattern: you don't fail at RAG, you fail at the seams between the pieces. Your chunking strategy doesn't match your embedding model's context window. Your retriever returns great results in a notebook and garbage in production because nobody wired up hybrid search. Your "agent" is really just a for-loop that stuffs retrieved text into a prompt and hopes. Foundry's whole pitch is that it owns those seams instead of leaving them to you. I was skeptical. I'm less skeptical now. What I Actually Did Step one: spin up a Foundry project. Not a hub-based one — those are legacy at this point, and if a tutorial has you creating one, skip it. The newer Foundry project type is the one to use. Step two: deploy two models. A chat model and an embedding model. Click, click, done. Both show up with their own endpoints. This part genuinely takes five minutes, and it's the first sign you're not building infrastructure anymore — you're configuring it. Step three: point Foundry at my documents. Blob storage in, Azure AI Search out. Foundry handles the chunking and embedding generation itself. I turned on hybrid search (keyword plus vector) because pure vector search on enterprise docs tends to miss exact terms people actually search for — product names, error codes, that sort of thing. If your content has a lot of that, don't skip this. Step four — and this is the part that's different from every tutorial I read two years ago. I didn't write a retrieval pipeline. I registered the search index as a tool on the agent and let the agent decide when to call it. Here's the whole thing: Python from azure.ai.projects import AIProjectClient from azure.identity import DefaultAzureCredential project = AIProjectClient.from_connection_string( credential=DefaultAzureCredential(), conn_str=os.environ["AIPROJECT_CONNECTION_STRING"], ) agent = project.agents.create_agent( model="gpt-4o-mini", name="docs-assistant", instructions=( "Answer only using retrieved context. " "Cite the source document for every claim. " "If the answer isn't in the retrieved content, say so." ), tools=[{ "type": "azure_ai_search", "index_connection_id": search_connection_id, "index_name": "example-index", }], ) thread = project.agents.create_thread() project.agents.create_message(thread.id, role="user", content="What's our refund policy for enterprise plans?") run = project.agents.create_and_process_run(thread.id, agent.id) No manual embedding calls at query time. No hand-written "retrieve top-k, stuff into prompt" logic. The agent framework does that internally, and it does it well enough that I stopped fighting it after the first try. Step five: For anything beyond simple lookups, I turned on agentic retrieval in Azure AI Search. Classic RAG fires one query per user turn, which quietly falls apart the moment someone asks a compound question — "compare our Q3 and Q4 policy and tell me what changed for renewals" is two questions wearing a trench coat. Agentic retrieval breaks that into sub-queries, runs them in parallel, and merges the results before generation. If your users ask messy, multi-part questions — and they do — turn this on from day one. Retrofitting it later is more annoying than it should be. Step six: Tested in the playground, then deployed the same agent behind a REST endpoint. Nothing about the agent changed between prototype and production. That alone would've saved me a full day on past projects. Now, the Part Everyone Skips I'm not going to pretend this is magic, because it isn't, and the tutorials that pretend otherwise are setting people up to get burned in a security review. Access control is on you. Foundry doesn't look at your documents and infer that HR files shouldn't be visible to the sales team. You configure document-level security filters in Azure AI Search yourself, and if you skip this, you've built a very articulate way to leak sensitive data. API keys are a prototype crutch, not a production plan. Move to Microsoft Entra ID before anything customer-facing goes live. This migration is a real afternoon of work, not a checkbox — budget for it. Retrieved documents are untrusted input. Prompt injection through a poisoned PDF is a real attack surface in every RAG system, Foundry included. Your system instructions need to assume the retrieved content might be trying to manipulate the model, because eventually it will. The costs stack. Embedding generation, index storage, and the extra tokens from stuffing retrieved passages into every call — none of this is free, and it compounds faster than people expect once you're past a demo and into real traffic. Model it before you commit to a chunking strategy at scale, not after. Was It Actually Worth It? Yes — but not for the reason most "look how easy this is" posts claim. The value isn't that RAG got simple. Grounding a model in the right data, with the right access controls, still takes real thought. The value is that Foundry took the boring week — the SDK wrangling, the manual retrieval loops, the glue code nobody wants to own — and turned it into an afternoon of configuration. That frees up the time you actually need for the parts that matter: is your data any good, is it chunked sensibly, and can you trust what comes back? If you've been putting off a RAG project because the infrastructure felt like too much, this is the moment to try again. Just don't skip the access control step to save time. That's the part that actually bites.

Artificial intelligence is rapidly transforming software testing by enabling QA engineers to generate test cases and test plans, automate browser interactions, analyze and debug failures, and execute complex testing workflows using simple natural-language prompts. While cloud-based AI assistants offer impressive capabilities, they often require subscriptions and sharing potentially sensitive application data with third-party services. Running an AI-powered testing assistant locally addresses these concerns by providing better privacy, lower operating costs, and complete control over the testing environment. In this tutorial, we’ll learn how to build our own local AI QA engineer using Docker, Ollama, Qwen3:8b, LibreChat, and Playwright MCP. It will allow us to perform browser automation and interact with web applications using natural language, all without relying on cloud-based AI services. Understanding the Architecture Every interaction begins with the user. For example, a user enters a prompt in LibreChat, such as “Open the Playwright website and click the ‘Get Started’ button.” LibreChat serves as the conversational interface through which users interact with the AI assistant. Rather than processing the request itself, it forwards the prompt to a locally hosted large language model, Qwen3:8b, running via Ollama. After receiving the prompt, Qwen3:8b interprets the user’s intent and generates a step-by-step execution plan. Instead of interacting with the browser directly, the model determines which tools are required and communicates those instructions using the Model Context Protocol (MCP). These MCP requests are handled by the Playwright MCP Server, which acts as the bridge between the language model and the browser. It translates the AI-generated instructions into executable Playwright commands. The Playwright MCP Server then launches a Chrome browser and performs the requested actions. Depending on the prompt, it can navigate to websites, click buttons, complete forms, extract text from web pages, capture screenshots, and execute a wide range of browser automation tasks. Once the browser completes the requested operations, the execution results are returned to Qwen3:8b. The language model analyzes the browser output and transforms the technical details into a clear, human-readable response. LibreChat then presents this response to the user. Instead of displaying raw Playwright logs, it provides a concise summary such as: “Navigation completed successfully. The Playwright website was opened, and the Get Started button was clicked successfully.” This architecture enables browser automation through natural language while ensuring that every component runs locally. As a result, we benefit from enhanced privacy, greater security, and complete control over the entire AI-powered automation workflow. Prerequisites Before getting started, ensure that the following software is installed on your machine: DockerNode.js 20 or higher versionGitOllama We’ll use Docker Desktop to run LibreChat, Node.js to install and run the Playwright MCP Server, Git to clone the required repositories, and Ollama to download and serve the local large language model. Having these tools installed beforehand will make the setup process smooth and straightforward. System Requirements Running a local AI-powered browser automation stack requires a reasonably capable machine. A system with 16 GB of RAM or more is recommended to run Docker containers and the language model efficiently. We’ll also need 20–25 GB of available disk space, preferably on an SSD, to accommodate Docker images and downloaded models. While a dedicated GPU can significantly improve model inference speed, it is entirely optional, and the setup works well on modern CPUs. For this tutorial, I’m using the following configuration: Operating system: macOS (M2 Pro)Memory: 16 GB RAM We can have the same setup on Windows and Linux, with only minor platform-specific differences in the installation steps. Setting Up the Environment for the Local AI QA Engineer Docker, Node.js, and Git are widely used development tools, and detailed installation guides for each are readily available online. Installing Ollama To install Ollama, either download the installer from the official website or use the installation command provided for your operating system. For macOS, it can also be installed using the following Homebrew command: Plain Text brew install ollama Once the installation is complete, it can be verified by running the following command in the terminal: Plain Text ollama --version Installing Qwen3:8b Qwen3:8b is chosen for this setup because it offers a strong balance of reasoning, code generation, and performance, making it ideal for Playwright TypeScript test generation, AI agents, MCP integration, and modern QA automation workflows while running efficiently on a local machine. However, other higher models can also be chosen if you know a better one. Another factor in choosing this model was the available system memory. Since my machine has 16 GB of RAM, some memory also needs to be reserved for other tools used in this setup, such as Docker, LibreChat, and Playwright. We need to start Ollama first by running the following command from the terminal. (It should be kept running in the background): Plain Text ollama serve Open a new terminal and run the following command to pull the Qwen3:8b model: Plain Text ollama pull qwen3:8b It should take some time to complete the pull, as the model is around 5.2GB. Once the download completes, we can check the model by running the command: Plain Text ollama list It should list the model downloaded. Next, we can quickly verify by running the model using the command: Plain Text ollama run qwen3:8b Once the model starts, it will prompt you to enter a query. To verify that everything is working correctly, try a simple prompt such as “What is 2 + 2?”. Observe how the model processes the request and generates its response. If the setup is successful, it should return the correct answer, 4, confirming that the model has been downloaded, installed, and is functioning properly. To stop the model, type “/bye” in the prompt, and it should exit. Qwen3:8b provides a good balance between performance and resource usage, making it a suitable choice for this hardware configuration. If more RAM is available, you can opt for larger LLMs that offer stronger reasoning and coding capabilities. Installing LibreChat With Docker LibreChat is an open-source AI platform that provides a unified and customizable interface for interacting with multiple AI models. It enables us to manage all our AI conversations from a single application while supporting features such as AI agents, Model Context Protocol (MCP) servers, custom tools, and integrations with both local and cloud-based LLMs. LibreChat acts as the front-end chat interface that communicates with the locally running Qwen3:8b model through Ollama. It allows us to execute AI-powered browser automation workflows entirely on our local machine. Follow the steps below to install LibreChat: Step 1: Clone the LibreChat GitHub Repository The repository can be cloned by running the following command: Plain Text git clone https://github.com/danny-avila/LibreChat After cloning the repository, navigate to the LibreChat folder, copy the .env.example file, and create a new .env file from it. Plain Text cd LibreChat cp .env.example .env Let's keep the .env file as it is, using the default values. Step 2: Connect Ollama to LibreChat Ollama can be connected to LibreChat by updating its configuration in the “librechat.yaml” file. The example file is already available in the cloned repo. Run the following command to copy librechat.example.yaml and create librechat.yaml. Plain Text cp librechat.example.yaml librechat.yaml Update the following configuration in the file to connect Ollama to LibreChat: YAML endpoints: custom: - name: "Ollama" apiKey: "ollama" baseURL: "http://host.docker.internal:11434/v1" models: default: - "qwen3:8b" fetch: true titleConvo: true titleModel: "current_model" summarize: false summaryModel: "current_model" modelDisplayLabel: "Ollama" Make sure that this configuration is added to the “custom” block, which falls under the “endpoints” block. This configuration adds Ollama as a custom AI endpoint in LibreChat. The baseURL tells LibreChat where to connect to the Ollama API, while the default model specifies that Qwen3:8b should be used by default. Since LibreChat is running inside a Docker container while Ollama is running directly on the host machine, we use http://host.docker.internal:11434/v1 instead of localhost. The special hostname host.docker.internal allows the Docker container to access services running on the host system, enabling LibreChat to connect to the locally running Qwen3:8b model through Ollama. Setting fetch: true allows LibreChat to automatically detect and display all models available in Ollama. The remaining options configure the user interface by generating conversation titles using the current model, disabling conversation summarization, and displaying the endpoint with the label Ollama in the LibreChat interface. Step 3: Mount the Configuration in the docker-compose-override.yml The docker-compose-override.yml can be copied and created in the same way as we did “librechat.example.yaml”. Plain Text cp docker-compose.override.yml.example docker-compose.override.yml The following block should be updated in the docker-compose.override.yml file. YAML services: api: volumes: - ./librechat.yaml:/app/librechat.yaml This file mounts the custom “librechat.yaml” configuration file into the LibreChat container. By mapping ./librechat.yaml to /app/librechat.yaml, Docker ensures that LibreChat uses the custom configuration each time the container starts. This approach allows us to modify settings, such as custom endpoints and AI models, without rebuilding the Docker image. Step 4: Start the LibreChat Application Using Docker Compose The LibreChat application can be started using the following command: Plain Text docker compose up -d It will take some time for the Docker images to download, and containers will start. Run the following command from the terminal to check the Container status: Plain Text docker ps -a This command displays the status of all Docker containers. If any container is unhealthy or encounters an issue, its status will be clearly indicated in the output. In case any container is unhealthy or encounters an issue, the following command can be run to check its logs: Plain Text docker logs <container name> Once all the containers are started successfully, open a new browser and navigate to http://localhost:3080 to start LibreChat. Since we are accessing LibreChat for the first time, we will be prompted to register and create a new user account. After completing the registration process, we can sign in and start using the application. Step 5: Selecting Ollama > Qwen3:8b Model By default, the gpt-5.5 model is selected. To select the Qwen3:8b model: Click on the gpt-5.5 modelSelect Ollama > Qwen3:8b Once the Qwen3:8b model is selected, we can verify if it is working by sending a simple prompt such as “What is 2+2?” Make sure the command “ollama serve” is already running in the terminal in the background, else the model Qwen3:8b won't work on LibreChat. Once we receive a successful response from the model, we can confirm that the Qwen3:8b model has been configured and integrated successfully with LibreChat. Install Playwright MCP Server Playwright MCP can be installed by running the following command in the terminal: Plain Text npx @playwright/mcp@latest \ --host 0.0.0.0 \ --allowed-hosts "*" \ --port 8931 \ By default, Playwright MCP listens only on localhost, which means applications running inside Docker (like LibreChat) cannot connect to it. Using --host 0.0.0.0 makes the server accessible from Docker containers, while --allowed-hosys "*" allows requests from host.docker.internal instead of restricting access to localhost. Once the Playwright MCP server is started, we can leave it running in the terminal. After the Playwright MCP server starts, it shows the following message at the bottom: “For legacy SSE transport support, you can use the /sse endpoint instead”. We will configure the Playwright MCP server using the SSE (Server-Sent Events) transport. Although Playwright MCP also supports the Streamable HTTP transport, LibreChat currently does not support connecting to it via the /mcp endpoint. Therefore, the SSE transport is used to establish a reliable connection between LibreChat and the Playwright MCP server. Configure Playwright MCP Server in LibreChat Playwright MCP server can be added to LibreChat by updating the following configuration in the “librechat.yaml” file. YAML mcpServers: playwright: type: sse url: http://host.docker.internal:8931/sse timeout: 120000 This configuration registers the Playwright MCP server with LibreChat. The type: sse setting specifies that the connection uses the Server-Sent Events (SSE) transport, while the url points to the Playwright MCP server running on the host machine. The hostname host.docker.internal allows the LibreChat Docker container to communicate with services running outside the container. The timeout: 120000 sets the request timeout to 120 seconds, giving the AI agent sufficient time to complete browser automation tasks before the connection expires. However, the timeout can be extended to 15–20 minutes or more, as there is no harm in doing that. YAML mcpSettings: allowedDomains: - 'host.docker.internal:8931' - 'localhost:8931' The mcpSettings configuration also needs to be added under the ‘actions’ block in the “librechat.yaml” file. The mcpSettings.allowedDomains section defines the list of trusted MCP server endpoints that LibreChat is allowed to connect to. By including both host.docker.internal:8931 and localhost:8931, LibreChat can establish a secure connection to the Playwright MCP server, whether it is accessed from within the Docker container (host.docker.internal) or directly from the host machine (localhost). Any MCP server not included in this list will be blocked, providing an additional layer of security. Restart the LibreChat app so it reads the newly configured Playwright MCP server: Plain Text docker compose restart That, or we can also shut down the already running LibreChat and start it again by using the commands below: 1. To shut down LibreChat: Plain Text docker compose down 2. To start it again: Plain Text docker compose up -d After restarting LibreChat, log in and navigate to the home page, and follow the steps below: Click on the MCP Settings menu on the left-hand menu panel.In the MCP Settings window, click on the “+” button to add MCP. Fill in the details for adding the Playwright MCP server; make sure to add the following settings: MCP server URL: http://host.docker.internal:8931/sseTransport: SSEAuthentication: NoneTick the “I trust this application” checkbox. Click on the “Create” button to save the details. Make sure that the Playwright MCP server is started and running on the terminal as discussed in the earlier section Click Connect for the newly created MCP server to establish the connection and begin using it. If everything is fine, a message should be displayed on successful connection. Understanding Model Context Protocol (MCP) By itself, a large language model (LLM) is limited to generating text. It can answer questions, explain concepts, write code, or summarize information, but it cannot directly interact with external systems or perform real-world actions. Model Context Protocol (MCP) changes this by enabling AI models to communicate with external tools and services through a standardized interface. Instead of simply providing suggestions, an AI model can execute tasks such as interacting with browsers, reading files, querying databases, or creating pull requests. Think of MCP as USB for AI A simple way to understand MCP is by comparing it to the USB standard. Before USB became the universal standard, every hardware manufacturer used its own proprietary connector. Printers, keyboards, cameras, and other peripherals all required different cables and custom software integrations. This made connecting devices unnecessarily complicated. USB solved this problem by introducing a common communication standard. Once both the computer and the device supported USB, they could communicate regardless of the device type. Whether you connected a keyboard, webcam, microphone, or external hard drive, the same protocol handled the communication. MCP brings the same level of standardization to AI systems. Without MCP, every AI application requires building and maintaining custom integrations for every external tool it wants to use. If we switch to a different AI application, those integrations often need to be recreated from scratch, resulting in duplicated effort and increased maintenance. A collection of awesome servers for the Model Context Protocol can be found at mcpservers.org. With MCP, tools expose a common interface that any MCP-compatible AI application can use. The AI model only needs to understand the MCP protocol, while the implementation details are handled by the individual MCP servers. Why MCP Matters for QA Automation For QA Automation Engineers, MCP unlocks the ability to automate complete testing workflows rather than isolated tasks. Consider the following request: “Read the Jira story, generate Playwright tests, execute them, analyze any failures, and create a GitHub pull request.” With MCP, the AI agent can coordinate multiple tools to complete the entire workflow. For example, it can: Read the user story from JiraAccess the application’s source code from GitHubGenerate Playwright TypeScript testsExecute the tests in a real browserCapture screenshots, logs, and execution reportsCommit the generated tests to GitHubUpdate the Jira ticket with the test results Each of these actions may be handled by a different MCP server, such as a Jira MCP server, GitHub MCP server, and Playwright MCP server. From the AI model’s perspective, however, every server is accessed using the same standardized MCP protocol. This standardization is what makes MCP so powerful. Rather than building custom integrations for every tool, AI systems communicate through a single, consistent protocol. As a result, MCP servers for Playwright, GitHub, databases, and many other services can be integrated and used in a uniform, scalable manner, significantly simplifying the development of AI-powered automation workflows. Creating an AI Agent With Playwright MCP Server in LibreChat for Automation Testing Let’s create a new AI Agent for browser automation testing with Playwright MCP using the steps below: Step 1: Click on the Agent Builder menu on the left-hand menu panel. Step 2: Enter the following mandatory details to create a new agent: Name: Provide a meaningful name to the agent.Category: Provide a category to the agent.Model: Select Qwen3:8bMCP Servers: Click on the Add MCP Server Tools button > Select the Playwright MCP Server that we created in the earlier section.Click on the Save button. Step 3: Update the model parameters. Clicking on the Model field, which has Qwen3:8b selected, should open the Model Parameters page. The following parameters can be set using this page: Provider: OllamaModel: Qwen3:8bTemperature: 0.2Top P: 0.85Frequency Penalty: 0.00Presence Penalty: 0.00Reasoning Effort: MediumReasoning Summary: Auto Click on the Save button to set the parameters. Step 4: Setting the instructions for the AI agent. The Following instructions can be pasted into the Instructions field in the Agent Builder window, or a “SKILL.MD” file can be created and uploaded using the Skills section of this agent. Markdown # Skills for the Local AI Agent for automation testing You are an expert QA Automation Engineer controlling a browser through Playwright MCP. Your goal is to execute browser actions safely and reliably. ## Tool Usage Rules - Do not run all MCP tools at the same time - Use only one Playwright MCP tool at a time. - Wait for the result of each tool before deciding the next action. - Never assume the page state. - Inspect the current page before interacting. - Do not start the next MCP tool unless the first one is complete ## Navigation Rules Treat the following actions as navigation-triggering actions: - Clicking Login, Submit, Continue, Save, Next, Checkout, etc. - Clicking any hyperlink. - Form submission. - Any action that changes the URL or reloads the page. - Wait until the page is fully loaded before making another tool call. After any navigation-triggering action: 1. Do not call any DOM inspection tool immediately. 2. Wait until the page has completely loaded. 3. Wait for the URL to stabilize if it changes. 5. Continue only after the new page is available. 6. Never inspect the previous page after navigation. ## Rules for locating web elements - Take a fresh snapshot to inspect the current page - Do not use XPath locator strategy - Use the same field name to locate elements, do not hallucinate and add prefix or suffix to field names - Use Semantic locator strategy: getByRole, getByText, getByLabel, getByPlaceHolder, getByAltText, getByTitle, getByTestId - Never use brittle CSS selectors such as .btn-primary, .container > div:nth-child(2), #content div span, or auto-generated classes. - Avoid nth() unless there is no unique locator. ## Interaction Rules - Verify and confirm that an element exists before interacting. ## Error Recovery If any Playwright tool fails: - Stop issuing new actions. - Inspect the current page. - Check Interaction Rules - Determine whether navigation has occurred. - Retry only if the page state confirms it is safe. - Do not repeat the same action more than once without confirming that the page state has not changed. Never repeat the same click more than once without checking the current page. ## Important If a click causes navigation, always assume the previous execution context has been destroyed. Do not read the DOM until the new page has fully loaded and a fresh snapshot has been obtained. Show a summary of test execution with the step count and pass or fail status - Run only the steps that are provided; do not hallucinate - Any deviation from these rules is not acceptable - Do not generate any additional steps - Always prioritize stability over speed. Providing instructions to an AI agent helps define its behavior, responsibilities, and the boundaries within which it should operate. These instructions act as persistent guidance, ensuring the agent follows consistent practices every time it performs a task instead of relying solely on the user’s prompt. For detailed setup instructions and troubleshooting guidance, refer to the GitHub repository. With these steps, the local AI agent is now ready to take commands. Running the AI Agent for Browser Automation To start using the AI Agent, click on New Chat.Click on the model name dropdown and select My Agents > The name of the agent that you created. Let’s use the following simple prompt and see how it works. Plain Text open http://playwright.dev verify the page title Once the prompt is submitted, we can observe the browser as the AI agent begins executing the task. The agent invokes the Playwright MCP server, which automatically launches a browser and performs the requested actions to navigate to the website and interact with the page. After the task is completed, Qwen3:8b analyzes the outcome and returns the results directly in the LibreChat conversation, demonstrating browser automation powered by Playwright MCP and Qwen3:8b. Let’s run another prompt for a login test scenario: Plain Text Navigate to https://parabank.parasoft.com/parabank/index.htm Locate "Username" field using "name=username" Enter "john" into the "Username" field. Locate "Password" field using "name=password" Enter "demo" into the "Password" field. Locator "Log In" button using "input[type="submit"] Click on the "Log In" button Verify that the "Accounts Overview" page is displayed This prompt also takes some time to understand the request before execution begins. It is important to note that the clearer and more specific the prompt, the more efficiently the AI agent can interpret and execute it. Well-structured prompts reduce ambiguity, minimize the chances of hallucinations, and typically result in faster execution and more accurate outcomes. As a best practice, break complex tasks into clear, sequential instructions whenever possible to improve the agent’s reliability and overall performance. As shown in the screenshot above, the AI agent invoked five tools from the Playwright MCP server to interact with the application and complete the requested workflow. It navigated to the website, located the username and password fields, entered the provided credentials, and submitted the login form. Finally, it verified that the login was successful by confirming that the “Accounts Overview” page was displayed. Since this setup runs entirely on a local machine, the AI agent takes approximately one minute to begin execution and around 4–5 minutes to complete a simple scenario. For more complex scenarios involving multiple steps, validations, or integrations, the AI agent is expected to take longer to analyze the request and complete the execution. But Execution time can be significantly reduced by running the setup on a machine with more powerful hardware, such as additional RAM, a faster CPU, or a dedicated GPU. Watch the step-by-step YouTube tutorial for Building your Local AI QA Engineer. Final Words Building a local AI QA engineer with Docker, Ollama, LibreChat, and Playwright MCP is an excellent way to explore the future of AI-powered software testing while keeping complete control over the data and infrastructure. By running everything locally, we eliminate recurring API costs, improve data privacy, and create a flexible environment for experimenting with AI-assisted browser automation using natural language. This setup is only the beginning of what’s possible. As we become more familiar with MCP and AI agents, the local QA assistant can be extended by integrating tools such as GitHub, Jira, databases, or custom MCP servers to automate even more of the testing workflow. Happy AI-powered testing!!

By Faisal Khatri CORE

Building a single AI agent is not usually the hard part. You send a prompt to a model, get a response back, and wire it into your app. Done. The hard part starts when that agent becomes one step in a larger system. A real AI workflow might need to ingest a file, extract text, chunk it, generate embeddings, call an LLM, write results to a database, sync to an external API, and notify a user. Those steps do not behave the same. Text extraction might finish in seconds. An LLM call might take minutes. A sync job might fail because some external API is having a bad day. That is where a lot of "agent" systems stop looking magical and start looking like regular distributed systems. I have seen this fail in boring ways: The same job gets processed twice.A worker writes to the database, then crashes before marking the job complete.A model call runs longer than expected and the message gets picked up again.A retried tool call creates duplicate external writes.Failed jobs sit in processing until someone manually checks the database. None of this is new. AI agents do not magically avoid old infrastructure problems. They still need queues, retries, idempotency, durable state, and monitoring. AWS SQS is a good fit for that middle layer. It is not a full workflow engine. I would not use it for every orchestration problem. But if you need a durable queue between independent agent stages, SQS is simple, reliable, and usually enough. The Coordination Problem A basic multi-stage AI workflow often looks like this: Plain Text Input source -> ingestion -> processing -> generation -> sync The first version is usually a database table with a status column. That works for a while. Then concurrency shows up. Two workers read the same pending row. A process crashes and leaves a job stuck in processing. Someone adds sleep(30) because the previous step "usually finishes by then." That last one is the kind of fix that works just long enough to become a production bug. A queue gives each stage a cleaner boundary. One stage publishes work. Another stage consumes it. If the next stage slows down, the queue absorbs the backlog instead of forcing the whole pipeline to wait. Plain Text Input Source -> ingest_queue -> Ingestion Worker -> chunk_queue -> Chunking Worker -> embedding_queue -> Embedding Worker -> summary_queue -> Summary Worker -> sync_queue -> Sync Worker Now ingestion can scale separately from summarization. If LLM generation is slow, messages pile up in summary_queue. That is not automatically a failure. That is what the queue is there for. A failed summary worker does not corrupt the whole workflow. The message can be retried. If it keeps failing, it moves to a dead letter queue. Standard Queues vs. FIFO Queues SQS gives you two main queue types: standard queues and FIFO queues. Standard Queues Standard queues give at-least-once delivery and best-effort ordering. A message can be delivered more than once. Messages may not arrive in the exact order sent. That sounds scary, but most background AI work should already handle this. Use standard queues for work like document processing, embedding generation, batch classification, independent user requests, and webhook processing. For these jobs, throughput matters more than strict ordering. FIFO Queues FIFO queues preserve ordering within a MessageGroupId and support deduplication. Use when sequence actually matters: conversation turns, per-user workflows, ordered state transitions. Python response = sqs.send_message( QueueUrl=queue_url, MessageBody=json.dumps(payload), MessageGroupId=payload["user_id"], MessageDeduplicationId=payload["task_id"] ) Be careful with the group ID. If every message uses the same MessageGroupId, you have serialized the whole queue by accident. Give each conversation, user, or workflow its own group ID so you preserve ordering per entity while allowing parallelism across different ones. My default rule: start with standard queues unless ordering is clearly required. Then make the handler idempotent. That matters more than the queue type. Ensuring Idempotency in Your Agent Flow Idempotency means the same task can run more than once without creating duplicate or incorrect side effects. This is the part I would not skip. SQS standard queues use at-least-once delivery, so duplicates are part of the contract. But this matters even more with AI workloads because model calls are expensive and outputs can be non-deterministic. Retrying the same prompt may cost money and return a different answer. Retrying the same tool call may send a duplicate email or write a second database row. The basic pseudo workflow: Plain Text receive message check if task already completed if completed, delete message and exit if not completed, process task store result delete message Simple version: Python def handle_message(message, store, sqs, queue_url): payload = json.loads(message["Body"]) task_id = payload["task_id"] if store.already_completed(task_id): sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "skipped", "task_id": task_id} result = run_agent_logic(payload) store.mark_completed(task_id, result) sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=message["ReceiptHandle"]) return {"status": "completed", "task_id": task_id} The store can be Postgres, DynamoDB, Redis, or anything durable with atomic writes. For Postgres, a unique constraint saves you: SQL CREATE TABLE agent_task_results ( task_id TEXT PRIMARY KEY, status TEXT NOT NULL, result JSONB ); INSERT INTO agent_task_results (task_id, status) VALUES ($1, 'processing') ON CONFLICT (task_id) DO NOTHING; If the insert succeeds, this worker owns the task. If it does nothing, another worker already claimed or completed it. The Failure Case I Designed Around Plain Text summary_queue -> Summary Worker -> Postgres -> sync_queue The summary worker receives a message, calls an LLM, writes the summary to Postgres, then deletes the SQS message. Now suppose the worker writes to Postgres but crashes before deleting the SQS message. From SQS's point of view, the job never finished. After the visibility timeout expires, another worker receives the same message and runs the task again. Without idempotency, that retry may call the LLM again, generate a slightly different summary, and write a second result. A safer handler checks whether model output already exists before calling the model: Python def summary_handler(payload, store): task_id = payload["task_id"] existing = store.get(task_id) if existing and existing.get("model_output"): summary = existing["model_output"] else: text = load_text(payload["input"]["text_uri"]) summary = call_llm(text) store.save_model_output(task_id, summary) store.save_final_result(task_id, {"summary": summary}) return {"next_stage": "sync", "next_input": {"summary_task_id": task_id} That avoids repeating the expensive part if the first attempt already got that far. Visibility Timeout When a worker receives a message, SQS hides it from other workers for the visibility timeout. If the worker finishes, it deletes the message. If the worker crashes, the message becomes visible again after the timeout expires. Too short: another worker receives the same message while the first is still running. Duplicate execution. Too long: failed jobs take too long to retry. Plain Text visibility_timeout = 2x to 5x expected processing time Reference: Metadata validation: 30-60 secondsEmbedding generation: 1-5 minutesLLM-heavy summary: 5-15 minutesLong document analysis: 15+ minutes with heartbeat For long-running tasks, extend visibility: Python sqs.change_message_visibility( QueueUrl=queue_url, ReceiptHandle=receipt_handle, VisibilityTimeout=extension_seconds ) The message should describe the work, not carry the workload. Bad: JSON {"task_id": "123", "full_pdf_text": "... thousands of lines ..."} Better: JSON { "task_id": "123", "stage": "summarize", "input": {"document_uri": "s3://bucket/docs/input.pdf"}, "metadata": {"user_id": "789", "priority": "normal"} } Store large files in S3. Send references through SQS. Do not let the queue become your storage layer. Dead Letter Queues A DLQ captures messages that fail repeatedly. Without one, poison messages cycle forever. Python sqs.set_queue_attributes( QueueUrl=main_queue_url, Attributes={ "RedrivePolicy": json.dumps({ "deadLetterTargetArn": dlq_arn, "maxReceiveCount": 5 }) } ) Use 3-5 as a starting point. A DLQ is not a trash bin - it's an alert. AI-Agent-Specific Failure Modes Duplicate LLM calls: Bigger bill, possibly different answer. Use task_id as idempotency key.Non-deterministic outputs: Store first successful output.Tool-call side effects: Make idempotent.Long-running inference: Use visibility heartbeat. What to Monitor MetricWhyApproximateAgeOfOldestMessageUser-facing delayApproximateNumberOfMessagesVisibleBacklogDLQ message countRepeated failures Two alerts: Oldest message exceeds latency targetDLQ has messages When SQS Is Not the Right Tool RequirementBetter fitSimple async tasksSQSVisual multi-step workflowStep FunctionsComplex event routingEventBridgeHuman approvalsStep Functions I have seen teams burn hours building multi-agent systems with database polling and sleep timers. It works at demo scale. It usually does not survive production traffic. SQS gives you durable message delivery primitives. But the app still needs idempotent handlers, visibility timeout tuning, and DLQ monitoring. Default architecture: One queue between major stagesStandard queues unless ordering requiredEvery handler idempotentLarge payloads outside the queueVisibility timeouts based on real processing timeDead letter queues for failures The difference between an AI demo and a reliable AI system is rarely the prompt. It is the infrastructure around the prompt. Build that layer intentionally.

If you've spent more than a year building enterprise Java apps, you've probably felt this specific kind of pain: a product manager asks for a new search filter, and you open your repository file to find it already has 18 methods. You write number 19, then 20, and somewhere around method 25 you start wondering if there's a better way. There is. It's called Spring Data JPA Specifications, and it's been sitting quietly in the framework the whole time. The Problem With Hard-Coded Query Methods Spring Data JPA's derived query methods are great for simple lookups. findByEmail is clean, readable, and requires zero SQL. But enterprise search rarely stays simple. Your CRM users want to filter customers by name and status. Then by date range. Then by city. Then by a keyword that could match name or email. Before long, you're maintaining a repository that looks like this: Java findByNameAndStatus(...) findByNameAndStatusAndCreatedDateBetween(...) findByNameOrEmailAndStatus(...) findByNameContainingIgnoreCaseAndStatusAndCreatedDateBetween(...) Each new requirement means a new method. The repository becomes a dumping ground. Testing it becomes a chore. Onboarding someone new becomes a conversation about which of the 30 methods to use. Specifications solve this by letting you define small, composable query predicates and combine them at runtime based on what filters the user actually provided. What a Specification Actually Is Under the hood, a Specification wraps the JPA Criteria API, the programmatic, type-safe way to build queries without writing raw SQL or JPQL. The Criteria API is powerful but verbose and tricky to read. Specifications give you that power with a cleaner surface area. Each Specification is just a lambda that produces a predicate: Java (root, query, criteriaBuilder) -> criteriaBuilder.equal(root.get("status"), "ACTIVE") That's it. One condition, one method, composable with anything else. Building It: A Customer Search Example Let's make this concrete. Imagine a Customer entity with name, email, status, and createdDate. Users can filter by any combination of these or none at all. The Entity Java @Entity public class Customer { @Id @GeneratedValue(strategy = GenerationType.IDENTITY) private Long id; private String name; private String email; private String status; private LocalDate createdDate; } A Specifications Utility Class Rather than scattering predicates across services, I keep them in a dedicated class: Java public class CustomerSpecifications { public static Specification<Customer> nameContains(String name) { return (root, query, cb) -> name == null ? null : cb.like(cb.lower(root.get("name")), "%" + name.toLowerCase() + "%"); } public static Specification<Customer> emailContains(String email) { return (root, query, cb) -> email == null ? null : cb.like(cb.lower(root.get("email")), "%" + email.toLowerCase() + "%"); } public static Specification<Customer> statusEquals(String status) { return (root, query, cb) -> status == null ? null : cb.equal(root.get("status"), status); } public static Specification<Customer> createdBetween(LocalDate start, LocalDate end) { return (root, query, cb) -> { if (start == null || end == null) return null; return cb.between(root.get("createdDate"), start, end); }; } } The null returns are intentional; Spring Data JPA ignores null predicates, which means you get automatic "skip this filter if not provided" behavior for free. The Repository Your repository needs to extend JpaSpecificationExecutor: Java public interface CustomerRepository extends JpaRepository<Customer, Long>, JpaSpecificationExecutor<Customer> { } Wiring It Together in the Service Java public List<Customer> searchCustomers(CustomerSearchRequest request) { Specification<Customer> spec = Specification .where(CustomerSpecifications.nameContains(request.getName())) .and(CustomerSpecifications.emailContains(request.getEmail())) .and(CustomerSpecifications.statusEquals(request.getStatus())) .and(CustomerSpecifications.createdBetween(request.getStartDate(), request.getEndDate())); return customerRepository.findAll(spec); } That single findAll call dynamically adapts to whatever combination of filters the caller provides. No branching logic, no 20 repository methods. When product asks for a fifth filter next sprint, you add one method to CustomerSpecifications and one .and() line in the service. Done. Going Further: OR Conditions, Joins, and Pagination OR Conditions The .or() combinator works exactly as you'd expect. A global search bar that checks name or email: Java Specification<Customer> spec = Specification .where(CustomerSpecifications.nameContains(keyword)) .or(CustomerSpecifications.emailContains(keyword)); Filtering Across Joins If your Customer has a nested Address, you can reach into it without any joins in your service layer: Java public static Specification<Customer> cityEquals(String city) { return (root, query, cb) -> city == null ? null : cb.equal(root.join("address").get("city"), city); } The join happens inside the Specification. Your service code stays clean. Pagination Because JpaSpecificationExecutor exposes a findAll(Specification, Pageable) overload, adding pagination is one line: Java Page<Customer> page = customerRepository.findAll(spec, PageRequest.of(0, 20, Sort.by("name"))); Mistakes I've Seen in the Wild Returning non-null predicates for null filters: This is the most common gotcha. If you forget the null check and return a valid predicate anyway, you'll silently filter out data that should be returned. Always guard at the top of the lambda. Mixing business logic into Specifications: A Specification should do one thing: produce a predicate. I've seen Specifications that log, that call services, that check permissions. Don't. Keep them pure. Creating a single "God Specification" that handles all filters: This trades the bloated repository problem for a bloated Specification problem. Small, single-purpose Specifications stay testable and reusable. A statusEquals Specification can serve your search screen, your reporting module, and your admin dashboard without any of them knowing about each other. Skipping case normalization for string searches: cb.like(root.get("name"), "%dzone%") won't match "DZone" or "DZONE." Always normalize: cb.lower(root.get("name")) paired with a lowercased input. Why This Pays Off Over Time The real dividend from Specifications shows up six months after you introduce them, when requirements change, and they always do. Adding a filter? One new static method, one .and(). Removing a filter? Delete the method and the combinator line. Reusing a filter across two features? Import the same Specification class. Unit testing a filter? Instantiate the Specification, pass a mock CriteriaBuilder, assert the predicate. No Spring context required. In complex enterprise codebases, the kind with multiple development teams, evolving product requirements, and a long maintenance tail, that kind of modularity is worth a lot more than it sounds at first. Final Thought Specifications aren't exotic. They're part of the Spring Data JPA standard library; they work with everything you already have, and they solve a problem that every team with a search screen eventually hits. If your repository is starting to look like an alphabetized index of every filter combination your users have ever requested, it's a good time to make the switch.

It was 2:14 in the morning when the pager went off. Our recommendation model's inference service had started returning 503s under a traffic spike that, frankly, wasn't even that big. Maybe three times the normal load. By the time I'd opened my laptop, the container had been OOM-killed four times in ten minutes, and Kubernetes was cheerfully restarting it into the same wall every ninety seconds. The image was 14GB. Cold start took eighty seconds. Nobody on the team had looked closely at any of that until it started costing us actual money in lost requests. That night is the reason I now have strong opinions about Docker and AI infrastructure. Why This Keeps Happening Containers became the default way to ship machine learning models because they solve a real problem: a model trained with CUDA 11.8, PyTorch 2.1, and a very specific glibc version doesn't reliably run on a colleague's machine, let alone a fleet of GPU nodes spread across three cloud regions. “It works on my machine” isn't a joke in ML infra; it's a recurring incident report. Docker gives you a way to freeze that dependency tree and ship it as one artifact, and that part genuinely works. What doesn't get discussed enough is the many issues that Docker does not resolve, along with the subtle ways teams exacerbate problems by forcing AI workloads into a packaging model that was originally designed for stateless web services. What We Tried First (and Why It Blew Up) Our first version of the inference image was, in hindsight, a small crime. We started from nvidia/cuda:12.2.0-devel-ubuntu22.04 because someone had seen it in a tutorial, installed the full CUDA toolkit, pip-installed every dependency without pinning, and copied in not just the model weights but three checkpoint versions “just in case.” Fourteen gigabytes. Every deployment pulled the entire image onto a fresh node, and during autoscale events, we experienced over a minute of waiting just for the image to be pulled before the container started loading the model into GPU memory. The first fix everyone reaches for is a multi-stage build, and yes, it helps — but it's not the silver bullet people pitch it as. Splitting a devel build stage from a runtime stage cut us from 14GB to roughly 6GB: Dockerfile FROM nvidia/cuda:12.2.0-devel-ubuntu22.04 AS builder WORKDIR /build COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt --target=/deps FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 COPY --from=builder /deps /usr/local/lib/python3.10/site-packages COPY model/ /app/model/ COPY serve.py /app/ WORKDIR /app CMD ["python3", "serve.py"] That's better, but the real lie in that Dockerfile is the COPY model/ line. We baked multi-gigabyte weights into an image layer, causing a full re-push of the weights with every code change, even minor ones, since Docker re-hashes the entire build context. We moved weights to an external volume pulled from object storage at container start, with a checksum cache to skip redundant downloads. That alone cut most of our deployment time. The trade-off is a slightly more complex startup script and a new dependency on storage being reachable at boot, which is its failure mode. There's no free lunch here; you're just choosing which problem is going to page you at 2 am. The Detour: Skipping Containers Entirely We also tried, briefly, running everything on bare metal with conda environments and skipping containers altogether, mostly because one engineer was convinced Docker added overhead with no real benefit for GPU workloads. It's not a crazy position; Docker's GPU story is genuinely leaky. <nvidia-container-toolkit exposes the host driver directly to the container, leading to potential mismatches that application-level packaging cannot resolve, so there is no real isolation>. However, we reverted that experiment within a sprint because as soon as more than two people interact with the training pipeline, environment drift reoccurs immediately. “Works on my conda env” is just “works on my machine” wearing a hat. What Actually Held Up in Production The architecture that worked was less about clever Docker tricks and more about admitting that an inference container and a training container have almost nothing in common and shouldn't share a Dockerfile, a registry strategy, or a deployment pattern. For serving, we kept images lean, stateless weights externalized, and a health assessment that actually runs a tiny dummy inference instead of just pinging an HTTP port. A server can report itself as “up” while a model failed to load correctly, and that gap has burned us more than once. Locally, a docker-compose GPU reservation block mirrored how production scheduled GPUs, so dev environments stopped lying about resource contention: YAML services: inference: image: registry.internal/rec-model:latest deploy: resources: reservations: devices: - capabilities: [gpu] count: 1 healthcheck: test: ["CMD", "python3", "healthcheck.py"] interval: 15s timeout: 5s retries: 3 For training, we accepted slower builds because training jobs run for hours, and a ninety-second image pull is negligible compared to that, even though the images are larger. Spending engineering time shrinking training images was effort we'd burned for no real payoff. A contrarian point I'll happily defend: not every container needs to be small, only the ones sitting in your hot path. The other decision that mattered more than any Dockerfile tweak was plain layer-caching discipline, putting "pip install before COPY." It sounds obvious when written down, but I've reviewed more than one ML team's Dockerfile that copies the whole repo first “for simplicity” and invalidates every cached layer on a README change. Key Takeaways Externalize model weights from the image; baking them in wrecks your caching and your deploy speed.Multi-stage builds help, but they don't resolve a runtime image that's still hauling around a full CUDA devel toolkit.GPU isolation via containers is partial; complete driver and toolkit mismatches between host and container remain entirely your problem.Training and serving images deserve different optimization priorities; don't apply the same size obsession to both.A health check that only confirms the process is running, without verifying that the model has loaded correctly, will mislead you at the most critical moments. Closing Thought Docker didn't fail us that night; rather, it was our incorrect assumptions about its free services that let us down. It's easy to treat containers as a solved problem because the tooling is so mature for web services and then be surprised when AI workloads expose every shortcut you took. I still think GPU container isolation lacks a clean solution, and I’m curious if upcoming tools will address it or if we'll continue improving our health-check scripts. What's the worst 2 am lesson your infrastructure has taught you?

Something I have noticed while talking to developers across different teams and projects is that almost everyone agrees that API testing matters. Almost everyone has an opinion on which framework is best. And almost nobody has a consistent, reliable API test suite that keeps up with their codebase. That gap between knowing testing matters and actually having good test coverage is where most of the interesting problems live. And a significant part of why that gap exists comes down to framework choices made for the wrong reasons, or made without enough information about what different frameworks actually do well. This article is about fixing that. Not by declaring one framework the winner but by giving you a clear picture of what each major option does, where it struggles, and how to match the tool to the actual problem you are trying to solve. What an API Testing Framework Actually Does Before getting into specific tools, it helps to be specific about what we are asking a framework to do. An API testing framework gives you a structured way to send HTTP requests to your API endpoints, define what a correct response looks like, assert that the actual response matches those expectations, and run all of this automatically as part of a broader testing pipeline. The better ones also handle setup and teardown of test state, organize tests into logical groups, produce readable output that makes failures easy to diagnose, and integrate cleanly into CI/CD pipelines so tests run on every commit without manual intervention. If you want a solid grounding in what is API testing in software before going further, that covers the conceptual foundation well. The rest of this article assumes you are past the basics and trying to make practical decisions about tooling. The Frameworks Worth Knowing REST Assured REST Assured is a Java library that provides a domain-specific language for writing REST API tests. If your backend is Java-based and your team is already living in the JVM ecosystem, it fits naturally. The fluent API reads well once you are used to it, and the JUnit and TestNG integration means tests slot into existing build and reporting infrastructure without extra configuration. Java given() .header("Authorization", "Bearer " + token) .contentType(ContentType.JSON) .body(requestPayload) .when() .post("/api/orders") .then() .statusCode(201) .body("orderId", notNullValue()) .body("status", equalTo("created")); The friction shows up when your team is not Java-heavy. REST Assured is verbose by design and the learning curve for developers coming from JavaScript or Python backgrounds is real. It also requires writing every test case manually, which means coverage depends entirely on what someone thought to write when the endpoint was built. Supertest Supertest is the Node.js answer to REST Assured. It wraps your Express or Fastify application and lets you send requests directly against it without spinning up a real server, which makes tests fast and removes network variability from the equation. Java const response = await request(app) .post('/api/users') .send({ email: '[email protected]', password: 'secure123' }) .set('Accept', 'application/json'); expect(response.status).toBe(201); expect(response.body.userId).toBeDefined(); The tight coupling to the application instance is both the strength and the limitation. Tests run fast because there is no real HTTP layer involved. But it means Supertest tests only cover your application in isolation and miss anything that happens at the network boundary or in downstream service interactions. pytest With requests or httpx Python teams have a genuinely good option here. pytest as a test runner combined with the requests library for HTTP is simple, readable, and flexible. httpx is worth knowing as a modern alternative that handles async APIs cleanly. Java def test_create_product(api_client, auth_headers): payload = {"name": "Widget", "price": 29.99, "stock": 100} response = api_client.post("/api/products", json=payload, headers=auth_headers) assert response.status_code == 201 data = response.json() assert data["name"] == "Widget" assert "productId" in data The pytest ecosystem is mature. Fixtures handle setup and teardown elegantly. Parametrize makes it straightforward to run the same test against multiple input variations. The main gap is the same one that affects all handwritten test frameworks: you get coverage for what you wrote tests for, not for how the API is actually used. Postman and Newman Postman deserves honest treatment here because it is genuinely useful and genuinely limited in ways that do not always get acknowledged. For manual exploration and sharing request collections across a team, Postman is excellent. The interface makes it easy to construct requests, inspect responses, and build up a library of examples that new team members can use to understand an API quickly. Newman, the CLI runner for Postman collections, lets you take those collections and run them in a CI pipeline. On paper, this sounds like a complete testing solution. In practice, the maintenance burden is significant. Every time an API changes, someone has to update the collection manually. In teams with frequent deployments, that process either happens consistently and consumes real engineering time, or it does not happen, and the tests drift silently from the actual API behavior. Postman works well as a development and documentation tool. It works less well as the primary automated testing infrastructure for a production API. Karate DSL Karate deserves a mention because it takes a genuinely different approach. Tests are written in a Gherkin-like syntax that is readable by non-developers, which matters in organizations where QA analysts or product people need to write or review tests without writing code. Java Feature: Order API Scenario: Create a new order successfully Given url 'https://api.example.com/orders' And header Authorization = 'Bearer ' + token And request { productId: '123', quantity: 2 } When method POST Then status 201 And match response.status == 'created' The tradeoff is that the DSL has its own learning curve and the abstraction that makes it readable also makes it harder to express complex test logic. It works well for straightforward functional tests and less well for tests that require significant setup, conditional logic, or deep integration with application code. Traffic-Based Test Generation This is a category rather than a single framework, and it represents a meaningfully different approach to the problem. Instead of writing test cases manually, tools in this space record real API traffic and generate test cases from observed behavior. The tests reflect how the API is actually being called rather than how someone anticipated it would be called when they were writing the spec. Keploy is worth understanding in this context. It sits in the request path, captures real API interactions, and generates regression tests from that traffic along with mocks for downstream dependencies. The practical advantage is that edge cases which only appear in real usage get captured automatically. The coverage grows with actual usage rather than with how much time someone had to write tests. This approach does not replace hand-written tests entirely. Tests that verify specific business logic or catch specific edge cases you know about still benefit from being written explicitly. But for building a baseline of regression coverage quickly, especially on an existing API that has thin test coverage, traffic-based generation addresses the problem in a way that traditional frameworks fundamentally cannot. The Dimension Most Comparisons Miss: Maintenance Over Time Most framework comparisons focus on writing tests. The more important question for a production system is what happens to those tests six months later. Hand-written tests have a maintenance cost that scales with both the size of the test suite and the rate of change in the API. Every breaking change requires someone to find the affected tests, understand why they are failing, and update them. In a team shipping multiple times a week, that is a recurring tax on engineering time. Self-healing capabilities, where a framework can detect that a change in the API caused a test to fail for structural reasons rather than behavioral ones and update the test automatically, exist in some commercial tools. For open-source frameworks, the maintenance is still largely manual. This is worth factoring into framework selection. A framework that makes writing the first test easy but creates significant ongoing maintenance burden may not be the right choice for a rapidly evolving API. How to Actually Choose The decision comes down to a few specific questions about your situation. What language does your team work in primarily? REST Assured for Java teams. Supertest for Node.js teams. pytest for Python teams. Crossing language boundaries creates friction that compounds over time. How often does your API change? A high change frequency pushes the decision toward frameworks with a lower maintenance burden or toward traffic-based generation approaches. Who needs to write and maintain tests? If the answer is only senior developers, any of the code-based frameworks work. If QA analysts or product people need to contribute, Karate or a tool with a visual interface becomes more relevant. What is the existing test coverage situation? Starting from scratch on a new API is a different context than trying to build coverage on an existing API that has been running in production for two years with real user traffic. Do you need mocks for downstream dependencies? If your API calls multiple external services, the friction of writing and maintaining mocks manually is substantial. Frameworks that generate mocks automatically from observed traffic change that calculation significantly. What Good API Testing Actually Looks Like The goal is not to choose the most popular framework or the one with the most GitHub stars. The goal is a test suite that catches real regressions reliably, runs fast enough to give developers feedback inside the CI pipeline, and does not require constant maintenance just to stay green. That combination is harder to achieve than it looks. The teams that get there are usually the ones that were honest about what their actual problems were before choosing a tool, matched the tool to the problem rather than the other way around, and treated test maintenance as a real ongoing cost rather than a one-time setup task. The framework is just the mechanism. The thinking that goes into what to test, how to organize tests, and how to keep coverage current as the API evolves is what actually determines whether the test suite is useful. Pick the framework that fits your context. Build the habit of treating test quality with the same seriousness as code quality. The coverage will follow.

Sponsored By: NutanixThe following is sponsored content. It may not reflect the views of our editorial staff. Most platform engineers are kept awake at night with some form of the same common complaint: the infrastructure bill does not align with what the infrastructure is really doing. For example, a GPU node pool provisioned for a monthly batch job might sit idle, burning budget for 20+ days out of 30. Or perhaps a business builds a standby data center designed specifically to account for a potential major outage, but that sits idle doing nothing every other day. CI/CD runners wait listlessly for the next pipeline trigger: fully provisioned, fully billed, but mostly idle. The above scenarios are nothing new. In fact, they could be considered the oldest problem in infrastructure: provisioning for peak, yet paying for average — or below. In modern technology, however, these pain points are felt mostly at scale. While enterprises spread Kubernetes® across hybrid clouds (on-premises clusters, cloud regions, and edge sites), idle capacity spreads beyond one team’s budget line. This spread compounds across every cluster that cloned or copied the same pattern of “just keep it running,” until it inevitably becomes a major structural cost that belongs to no one. So what is the solution? The assumption might be to contrive a smarter way of bin-packing always-on nodes. The real, better solution is to remove the notion of “always-on” entirely; instead, platform engineers should focus on building infrastructure that sits at true zero and becomes real, schedulable capacity only when required. Why does hot standby become a liability? The concept of over-provisioning is common, and it is a reasonable instinct that most platform engineers pursue. Teams that run batch jobs or use AI and ML pipelines often deal with expensive, hard-to-find resources (GPUs in particular), and the dreaded fear of a cold start leading to a delay in a critical job is all too real. The usual approach is to keep any needed resources reserved, even if the job requires them only once or twice in a given month. That basic instinct, when applied at the level of a whole data center, is the cause that results in the classic “hot standby” disaster recovery pattern: a fully provisioned secondary site that mimics (or mirrors) production, yet does not do real work, as it is waiting for a failover event that might never occur. It’s an expensive insurance policy that organizations hope never to use, and the expense often largely sits idle. As enterprises lean into AI and automated workloads, as well as retail and edge deployments, we see this pattern replicate further. Monthly recurring jobs don’t need their own dedicated GPU pool sitting idle for the rest of the month. Likewise, CI/CD pipelines don’t need agents that run around the clock to handle an occasional pull request. Multiply that behavior across every team, region, and cluster operating with the same "play it safe" mindset, and idle capacity stops looking like a rounding error. It becomes a real budget item. What appears to be a dozen reasonable decisions is, in reality, one costly pattern. That makes the root cause much harder to identify and resolve. While hot standby remains common for disaster recovery, the more immediate opportunity for Kubernetes platform teams is eliminating permanently provisioned node pools for intermittent workloads. Scale-from-zero applies the same principle — capacity exists only when demand requires it — but at the infrastructure layer where operational costs accumulate most quickly. How does scale-from-zero actually work? Scale-from-zero is easy to understand in practice, but a little more difficult to engineer. In essence, a node pool with zero running nodes should still be visible to the Kubernetes scheduler as “available capacity.” If the schedule cannot see it, it cannot plan for it, and a pod requesting resources from an empty pool will sit in a pending state until someone notices and takes steps to intervene. This is where capacity annotation becomes important. Instead of relying on the old approach of provisioning tiers (i.e.; the “gold, silver, and bronze” classes used to define the offerings of a node pool), Nutanix Kubernetes Platform (NKP) attaches metadata directly to the node pool definition, letting the scheduler know exactly what capacity would exist if a node were running CPU, memory, and GPU resources at a finer grain than “one whole GPU.” What is the Nutanix Kubernetes Platform? Nutanix Kubernetes Platform (NKP) is an enterprise Kubernetes platform that simplifies deploying, managing, and scaling fleets of Kubernetes clusters across hybrid and multicloud environments while giving organizations the flexibility of an open, Kubernetes-native architecture. Dynamic Resource Allocation for GPU workloads is a solid example of where this trend is heading. Instead of allocating your entire GPU to a job that only requires 50 GB of memory out of a 100 GB card, a scheduler can consider the actual resource needs and assign workloads in a more precise manner, even before provisioning any hardware. In both theory and practice, this means the scheduler can make a placement decision on a node pool with zero nodes running, treating the annotated capacity as though it were live, queuing the pod against the pool, and making decisions to trigger the autoscaler to actually provision resources. This is the part of the architecture where the real work occurs, behind the scenes. Cluster API (an open-source Kubernetes project for declarative cluster lifecycle management), with the Nutanix-specific provider acting as the implementor, gives IT teams a definitive way to define how a cluster or node pool looks. Cluster API then acts as the engine that drives the infrastructure to that state. What is CAPX? CAPX (the Cluster API Provider for Nutanix) is the component that translates Kubernetes infrastructure intent into concrete operations on Nutanix AHV, Nutanix's enterprise hypervisor. When Cluster API determines that a MachineDeployment needs additional capacity, CAPX reconciles that desired state by provisioning virtual machines, applying the appropriate templates and networking configuration, bootstrapping Kubernetes components, and registering the new node with the cluster. From the platform engineer's perspective, scaling remains declarative: the desired node count changes, while CAPX handles the infrastructure orchestration required to make that state a reality. CAPX allows admins to stand up Kubernetes clusters rapidly across multiple locations, including retail and AI edge deployment, providing an autoscaler that expands or shrinks the infrastructure automatically, without relying on someone to manually provision a VM at an inconvenient time. From an operator's perspective, the scaling workflow follows a predictable sequence: A workload is created with CPU, memory, GPU, or other scheduling requirements.The scheduler determines no existing node satisfies those requirements, leaving the pod in a Pending state.Cluster Autoscaler, the Kubernetes component responsible for adjusting node capacity based on pending workloads, evaluates the unschedulable workload and identifies a compatible scale-from-zero node pool based on its capacity annotations.Cluster API updates the corresponding MachineDeployment to request additional infrastructure.CAPX provisions the required virtual machine in Nutanix AHV, attaches networking, and performs the bootstrap process.The new node joins the Kubernetes cluster and reports a Ready status.Kubernetes binds the pending workload to the newly available node.Once demand subsides and scale-down thresholds are reached, the autoscaler removes the node and the pool returns to zero capacity. Logs, events, and the full picture It should go without saying that if the platform engineer and team can’t see it happening, then none of the above is useful. Kubernetes does give you basic autoscaling visibility out of the box, but it lacks the observability level many enterprises really need. To validate a correctly running scaling event, you need two sources in particular: Source What it tells you Cluster Autoscaler logs These show you the actual scaling decision, why it chose to scale a pool, the capacity calculation that triggered it, and whether it considered alternative pools first. Kubernetes event traces These showcase what happened to the workload and the node, such as pod scheduling outcomes, node registrations, and readiness condition transitions. When viewed together, these two sources reconstruct the full state-machine transition from a pod against a zero-capacity pool to a pod running on live infrastructure. This is where many real-world misconfigurations emerge. For example, an application may be deployed without resource limits. If a workload consumes more memory than anticipated, the autoscaler can mistakenly provision additional nodes, even though the underlying issue is resource allocation rather than insufficient capacity. The fix is straightforward: set resource limits. Yet under deadline pressure, this step is easy to overlook, and the resulting costs may not become apparent until they appear on a cloud bill. NKP enhances enterprise cloud-native security beyond native open-source tooling through strategic ecosystem integrations. By partnering with RapidFort for vulnerability management and Canonical to deliver Ubuntu Pro as a trusted, built-in base OS option, NKP is designed to provide a secure, resilient, and compliant foundation for production workloads. Beyond securing the platform itself, production Kubernetes environments require operational consistency and visibility to run at scale. GitOps, specifically FluxCD, helps keep the desired state of managed clusters reconciled against a single Git source of truth. In addition, observability tools like Prometheus Alert Manager deal with the notification layer, routing scaling events to communication avenues like Slack, Microsoft Teams, or SMS messaging. This provides platform engineers with better visibility into scaling events, reducing the need to examine logs to determine whether a workload scaled when it shouldn't have. Automating Node Pool Lifecycle Scale-from-zero becomes far more valuable when node pool configuration is automated rather than managed manually. As environments grow, editing individual MachineDeployment manifests quickly becomes difficult to maintain, particularly when GPU pools, edge clusters, and development environments all require different scheduling policies. NKP provides the operational workflow for creating and managing node pools. Rather than manually editing manifests, platform teams can inject labels, taints, and capacity annotations into MachineDeployment definitions as part of a repeatable automation pipeline before committing those changes through GitOps. GPU node pool configuration checklist As an example, a GPU node pool might receive: Labels: identifying the workload typeTaints: preventing general-purpose schedulingCapacity annotations: describing available GPU, CPU, and memory resources Because these changes are generated consistently through automation instead of manual editing, platform teams reduce configuration drift across clusters. Combined with GitOps reconciliation through FluxCD, the desired configuration remains version-controlled, repeatable, and significantly easier to audit as infrastructure evolves. The next shift What ties these approaches together — active-active architectures instead of hot standby, capacity annotations instead of static machine classes, and scale-from-zero node pools instead of permanently reserved infrastructure — is a shift away from provisioning for the worst-case scenario and toward provisioning for actual demand. Infrastructure is no longer something you size once and live with. Instead, it becomes an elastic resource that expands and contracts in response to real-world signals, such as pending pods, queued pipeline jobs, or traffic spikes. This broader focus on infrastructure automation also extends to initiatives such as Nutanix’s bare-metal deployment capability NKP Metal, which aims to reduce cluster deployment times, reinforcing the same operational principle: infrastructure should be provisioned quickly and only when required. None of this can replace or eliminate the need for engineering judgment. Deciding which workloads belong on scale-from-zero pools versus always-on infrastructure still requires careful evaluation of latency requirements, cold-start risk, and workload characteristics. However, the infrastructure debt created by defaulting to a "just keep it running" approach is no longer an unavoidable consequence of operating Kubernetes at scale. Increasingly, capacity can be provisioned only when demand requires it, allowing infrastructure to align more closely with actual workload needs. Visit Nutanix to learn more about how Nutanix Kubernetes Platform enables deterministic scale-from-zero, infrastructure elasticity, and automated node pool lifecycle management across hybrid cloud environments.

Modern CI/CD pipelines often execute complete regression suites for every code change, regardless of the actual impact of the modification. While this approach guarantees broad validation coverage, it also introduces unnecessary test execution, slower feedback loops, and increased infrastructure cost. This challenge becomes more visible in microservice-based systems where repositories, services, and automation suites are distributed across multiple projects. A small change in one module can unintentionally trigger an entire regression pipeline containing tests unrelated to the updated code. To explore a lightweight solution for this problem, I built a personal engineering project that performs impact-based test selection using Git diff analysis, Spring Boot, JGit, GitHub Actions, and Karate. The goal of the project was simple: instead of executing the full regression suite for every change, dynamically determine which tests are actually impacted and execute only those tests. The project demonstrates how selective test execution can help reduce unnecessary CI workload while maintaining baseline validation coverage through fallback smoke testing. The Problem With Traditional Regression Execution In many CI/CD pipelines, regression execution is static. Every push event or pull request triggers: Full API regression suitesComplete integration testingBroad validation across unrelated modules Although this guarantees high coverage, it creates several practical problems. Slow Feedback Cycles Developers may wait several minutes or even longer to receive pipeline feedback for small localized changes. For example, updating a single payments API controller might still trigger: transactions teststransfer testsauthentication regression testsunrelated smoke validations As projects scale, this delay affects engineering productivity and release speed. Unnecessary Infrastructure Usage Executing the same large regression suites repeatedly consumes unnecessary CI resources. This becomes more expensive when: Pipelines run in parallelMultiple pull requests are activeCloud runners are billed by execution time Reduced Pipeline Efficiency In many cases, only a small subset of tests is truly relevant to the code change. Running the full suite results in redundant execution and inefficient utilization of CI infrastructure. The objective of this project was to explore whether lightweight impact analysis could reduce unnecessary test execution without introducing complicated tooling or dependency management systems. Project Overview The solution uses Git diff analysis to identify changed files and map those changes to relevant Karate test tags. The architecture consists of two repositories: Plain Text Developer Repository (fintech-impact-services) ↓ Push Event GitHub Repository Dispatch ↓ Automation Repository (karate-change-impact-test) ├── Start Spring Boot Application ├── Perform Git Diff Analysis ├── Generate Impacted Test Tags ├── Execute Targeted Karate Tests └── Generate Execution Metrics The development repository contains the Spring Boot microservice and impact analysis API. The automation repository contains Karate test suites and GitHub Actions workflows responsible for selective test execution. This separation allowed the automation layer to remain reusable and independently managed. Cross-Repository Workflow The workflow begins when code is pushed to the development repository. A GitHub Repository Dispatch event triggers the automation repository pipeline. Example dispatch payload: Shell curl -X POST \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer <TOKEN>" \ https://api.github.com/repos/{owner}/karate-change-impact-test/dispatches \ -d '{ "event_type": "dev_push" }' The automation repository then performs the following steps: Checkout repositoriesStart the Spring Boot impact-analysis serviceWait for API readinessCall the impact-analysis endpointRetrieve impacted Karate tagsExecute only selected testsPublish execution metrics This design helped simulate a lightweight cross-repository CI orchestration model using only GitHub-native capabilities. Implementing Git Diff Analysis Using JGit The core functionality of the project relies on identifying changed files between commits. Instead of using shell-based Git commands inside CI scripts, the implementation uses JGit, a Java library that provides Git functionality directly within Java applications. The service compares the current branch with a target branch reference and extracts modified file paths. Example implementation: Java public Set<String> getImpactedTags(String targetBranch) throws Exception { FileRepositoryBuilder builder = new FileRepositoryBuilder(); Repository repository = builder.readEnvironment() .findGitDir(new File(".")) .build(); try (Git git = new Git(repository)) { AbstractTreeIterator oldTreeParser = prepareTreeParser(repository, targetBranch); List<DiffEntry> diffs = git.diff() .setOldTree(oldTreeParser) .call(); return calculateTags(diffs); } } Once the diff entries are collected, the application extracts changed paths and sends them to the mapping engine. Using JGit provided several advantages: Better portability across environmentsEasier integration with Spring BootReduced dependency on shell scriptingCleaner CI pipeline implementation One challenge encountered during implementation was ensuring full Git history availability inside GitHub Actions runners. Shallow clones occasionally caused incorrect branch comparisons, so complete fetch depth was required for accurate analysis. Rule-Based Impact Mapping After identifying changed files, the framework maps those files to relevant Karate execution tags. Example mapping rules: Changed PathKarate Tag/payments/@payments/transactions/@transactions/transfer/@transfers/auth/@regressionpom.xml@regression Example implementation: Java if (path.contains("/payments/")) { tags.add("@payments"); } if (path.contains("/transactions/")) { tags.add("@transactions"); } The project intentionally uses deterministic rule-based mapping instead of advanced dependency graph analysis or machine learning models. The primary reasons were: SimplicityPredictabilityEasier debuggingFaster implementationLower maintenance overhead Although more sophisticated impact-analysis systems exist, lightweight rule-based mapping was sufficient to demonstrate measurable CI optimization in this project. Dynamic Karate Test Execution Once impacted tags are generated, the automation repository dynamically executes only the relevant Karate tests. Example command: Shell mvn test -Dkarate.options="--tags @payments,@transactions" The Karate framework worked particularly well for this implementation because feature files were already organized by business capability. Example structure: Gherkin features/ ├── payments.feature ├── transactions.feature ├── transfers.feature └── smoke.feature This made selective execution straightforward without requiring custom runners or additional orchestration frameworks. Safe Fallback Mechanism One important concern with selective execution is the possibility of hidden dependencies. A code change may indirectly impact areas not captured by simple path-based rules. To reduce this risk, the framework includes a fallback strategy. If no impacted tags are identified, the pipeline automatically executes baseline smoke tests. Example: Shell @smoke This ensured that critical application validation still occurred even when no direct impact mapping was detected. The fallback mechanism helped balance optimization with CI reliability. GitHub Actions Integration GitHub Actions was used to orchestrate the complete workflow. Example workflow steps: YAML - name: Start Spring Boot Service run: mvn spring-boot:run & - name: Wait for API Readiness run: | curl --retry 10 \ --retry-delay 5 \ http://localhost:8080/actuator/health One practical issue encountered during implementation was synchronization between application startup and API invocation. Without readiness checks, the workflow occasionally attempted to call the API before the Spring Boot application was fully initialized. Adding retry-based health checks improved workflow stability significantly. Execution Metrics To evaluate the effectiveness of the approach, the framework generates execution metrics after each run. Example output: JSON { "impacted_tags": "@payments,@transactions", "scenarios_executed": 6, "scenarios_skipped": 12, "test_reduction_rate": "66%", "timing_metrics": { "total_workflow_seconds": 52, "isolated_test_seconds": 18, "api_overhead_seconds": 6 } } In sample execution scenarios from this project, the framework reduced executed tests by approximately 60–70% depending on the scope of code changes. Localized feature updates benefited the most, while shared dependency changes still triggered broader regression execution. Although these results came from a personal engineering project rather than a production enterprise system, the experiment demonstrated how lightweight impact-aware execution can improve CI efficiency. Limitations and Future Improvements The project also exposed several limitations. Manual Mapping Maintenance Rule-based mappings require periodic updates as repositories evolve. Hidden Dependency Risks Indirect service dependencies may not always be detected through simple path matching. Git Comparison Accuracy Accurate impact analysis depends heavily on proper branch comparison and repository history availability. Future improvements could include: dependency graph analysiscode coverage–based impact detectionhistorical test-failure analysisML-assisted impact predictionmulti-module dependency propagation These enhancements could improve precision while preserving the lightweight nature of the framework. The complete implementation for this project, including the Spring Boot impact-analysis service and GitHub Actions workflow, is available on GitHub. Source Code: Dev repository: https://github.com/raakeshdev20/fintech-impact-servicesAutomation repository:https://github.com/raakeshdev20/karate-change-impact-test Conclusion This project explored how Git diff analysis and selective test execution can help optimize CI/CD pipelines without requiring complex external tooling. By combining the following, the framework demonstrated a practical approach to reducing unnecessary regression execution for localized changes. JGit-based change detectiondeterministic impact mappingdynamic Karate executionGitHub Actions orchestration While the implementation is intentionally lightweight, the experiment highlights how impact-aware testing strategies can improve feedback cycles, reduce redundant execution, and make CI pipelines more efficient as systems continue to scale.