Summary
Adds a CPT-level (Testcontainers + real LLM) e2e test suite for the AI Agent connector's Google Vertex AI provider, mirroring the pattern established for the OpenAI provider in camunda/team-qa-engineering#653 / #6940.
This extends that same automated E2E testing initiative to a second provider.
What was implemented
PR: #8226
New test class GoogleVertexAiE2ETestIT with 4 real-LLM CPT scenarios:
- Happy path — single-turn date/time question, judged on the response containing the Berlin time.
- Explicit tool instruction (
shouldCompleteWithUserLookupTool) — tool lookup with an explicit instruction to use the tool.
- Natural-language tool inference (
shouldInferOrderStatusToolFromNaturalRequest) — tool lookup inferred from a natural request that never names a tool, against a new GetOrderStatus HTTP tool.
- Cross-turn context retention (
shouldRetainToolResultAcrossFeedbackLoop) — context retention across a feedback loop, with a rigor check that the tool only executes once, proving the follow-up answer came from retained conversation history rather than a fresh tool call.
Plus a @Disabled placeholder test for the Gemini 3 thoughtSignature fix, mirroring the OpenAI suite's pattern of staging tests ahead of provider readiness.
A new shared base class, AbstractAiAgentE2ETestIT, was extracted so the OpenAI and Vertex AI test classes share the same fixture setup, mocked-HTTP-tools handler, and tool result fixtures instead of duplicating them.
CI reuses the existing IDP_GCP_* Vault secrets already available to the connectors CI role, wired into the existing run-ai-agent-cpt job.
Links
Known open item
Full Gemini 3 coverage is blocked on #8178 (Vertex SDK migration to google-genai) landing on main. The disabled placeholder test is ready to be enabled once that PR merges.
Notes
Filed at the request of Gustavo Betances.
Generated by Claude Code
Summary
Adds a CPT-level (Testcontainers + real LLM) e2e test suite for the AI Agent connector's Google Vertex AI provider, mirroring the pattern established for the OpenAI provider in camunda/team-qa-engineering#653 / #6940.
This extends that same automated E2E testing initiative to a second provider.
What was implemented
PR: #8226
New test class
GoogleVertexAiE2ETestITwith 4 real-LLM CPT scenarios:shouldCompleteWithUserLookupTool) — tool lookup with an explicit instruction to use the tool.shouldInferOrderStatusToolFromNaturalRequest) — tool lookup inferred from a natural request that never names a tool, against a newGetOrderStatusHTTP tool.shouldRetainToolResultAcrossFeedbackLoop) — context retention across a feedback loop, with a rigor check that the tool only executes once, proving the follow-up answer came from retained conversation history rather than a fresh tool call.Plus a
@Disabledplaceholder test for the Gemini 3thoughtSignaturefix, mirroring the OpenAI suite's pattern of staging tests ahead of provider readiness.A new shared base class,
AbstractAiAgentE2ETestIT, was extracted so the OpenAI and Vertex AI test classes share the same fixture setup, mocked-HTTP-tools handler, and tool result fixtures instead of duplicating them.CI reuses the existing
IDP_GCP_*Vault secrets already available to the connectors CI role, wired into the existingrun-ai-agent-cptjob.Links
agentic-ailabel to trigger AI Agent CPT tests on a PR before merge queue #8234Known open item
Full Gemini 3 coverage is blocked on #8178 (Vertex SDK migration to
google-genai) landing onmain. The disabled placeholder test is ready to be enabled once that PR merges.Notes
Filed at the request of Gustavo Betances.
Generated by Claude Code